When something fails,
what survives?
Durable events, recoverable leases and bounded retries are useful only when their failure paths are exercised.
8 recorded cases. All passed.
Recorded 2026-11-04T09:13:50.030Z · source: docs/RELIABILITY.md
- 01
Postgres unavailable: outbox claim fails without acknowledging work, then recovers
Observed result: PASS
188.0 ms
- 02
Postgres connection cut mid-fanout rolls back dedupe, delivery and acknowledgement together
Observed result: PASS
122.6 ms
- 03
Pool exhaustion has a bounded acquisition failure and no delayed phantom claim
Observed result: PASS
248.3 ms
- 04
SIGKILL after outbox claim: expired lease is reclaimed once and stale owner is rejected
Observed result: PASS
196.4 ms
- 05
FCM outage: backoff prevents immediate retry and provider recovery records one delivery
Observed result: PASS
124.3 ms
- 06
Persistent FCM outage stops at retry limit and dead-letters the job
Observed result: PASS
156.2 ms
- 07
SIGKILL after simulated FCM acceptance: retry preserves event identity and unique row
Observed result: PASS
226.8 ms
- 08
Redis unavailable: OTP checks reject; recovered store retains one-time verification
Observed result: PASS
114.3 ms
Durations are test execution times, not endpoint latency measurements.
What was actually exercised.
- Real local PostgreSQL and Redis operations with test-owned connection failures.
- Service subprocesses killed with SIGKILL, followed by lease-recovery checks.
- Injected transient and persistent FCM failures, including retry and dead-letter behavior.
- Private fixture schemas and exact cleanup of test-owned resources.
The limits matter too.
These tests did not stop database daemons, reboot a host or contact live FCM. They did not test the complete production supervisor loop.
Fixtures copied tables, checks and indexes. This M8.1 harness did not copy foreign keys or triggers and did not replay all migrations. It is not the deferred M8.4 clean-database suite.
Notification delivery remains at least once. If a provider accepts a send before the worker crashes, retry can repeat the external send. Stable event identity and one database row do not prove exactly-once delivery to a handset.