Engineering notes / 02 — Open-source lab
Spectra
Durable workflows. Human control.
A Python control plane and Go workers for approval-gated security workflows, crash recovery, and auditable execution.
- Python
- Go
- React
- TypeScript
- PostgreSQL
- SQLite
- Docker
The engineering problem
A worker can crash after an external action succeeds but before the system records completion. Retrying blindly risks repeating a sensitive action. Spectra makes that failure window explicit and keeps authorization separate from execution.
Follow the request.
- 01
Alert & investigation
Tenant-bound evidence and bounded read tools
- 02
Policy & approval
Validate target and evidence; require approver role
- 03
Durable queue
Transactional jobs, fenced leases, bounded retries
- 04
Execute & reconcile
Lookup external action, verify effect, record timeline
02 / Design decisions
Where the architecture earns its keep.
Make recovery part of the workflow
Stable external action keys and provider lookup allow a replacement worker to reconcile an existing effect. Fenced leases and heartbeats coordinate ownership; retries are bounded and jittered.
Constrain agent authority
The optional investigator has typed, tenant-bound read tools, a step budget, and structured proposals. Sensitive actions still pass independent evidence and target validation followed by human approval.
Leave an inspectable trail
Persisted execution timelines expose the workflow in React. Application events support per-run hash-chain verification, with Prometheus metrics and optional API tracing for operational visibility.
03 / Verification & scope
Failure is part of the specification.
The local lab exercises recovery after isolation succeeds and the worker dies before acknowledgment. Go race checks, contract tests, and recovery flows cover the real queue, approval, database, and HTTP paths.