Engineering notes / 04 — Open-source library
RetryCheck
Commit. Crash. Retry. Verify.
A Python library for testing business effects under duplicate delivery, concurrent retries, and worker crashes after commit.
- Python
- pytest
- SQLite
- HTTP
- Multiprocessing
The engineering problem
The dangerous gap is after an operation commits and before the caller receives acknowledgment. A retry can duplicate a charge, receipt, or record unless the operation and its idempotency strategy survive that gap.
Follow the request.
- 01
Snapshot state
Capture the business effects that matter
- 02
Execute operation
Commit over real HTTP or a local callback
- 03
Inject failure
Lose a response, race delivery, or crash a worker
- 04
Retry & assert
Compare effects and inspect attempt timelines
02 / Design decisions
Where the architecture earns its keep.
Test effects, not just responses
Explicit before/after snapshots and caller-defined assertions verify business outcomes. Structured attempt timelines retain the evidence needed to understand what each delivery did.
Control the failure window
Ordered barriers exercise concurrent delivery without depending on a lucky scheduler. Process restart scenarios use spawned workers to test a crash after commit and before acknowledgment.
Separate bugs from broken tests
Verification failures are distinct from operation, client, snapshot, and infrastructure errors. The included HTTP and SQLite example contrasts unsafe duplication with a transaction-backed idempotent operation.
03 / Verification & scope
Failure is part of the specification.
Scenarios cover sequential and concurrent duplicates, response loss after commit, and worker crash/restart. The example uses real HTTP and SQLite, while the library keeps its runtime dependencies in the Python standard library.