Download the recorded evidence
← Engineering

When something fails,
what survives?

Durable events, recoverable leases and bounded retries are useful only when their failure paths are exercised.

8 recorded cases. All passed.

Recorded 2026-11-04T09:13:50.030Z · source: docs/RELIABILITY.md

  1. 01

    Postgres unavailable: outbox claim fails without acknowledging work, then recovers

    Observed result: PASS

    188.0 ms

  2. 02

    Postgres connection cut mid-fanout rolls back dedupe, delivery and acknowledgement together

    Observed result: PASS

    122.6 ms

  3. 03

    Pool exhaustion has a bounded acquisition failure and no delayed phantom claim

    Observed result: PASS

    248.3 ms

  4. 04

    SIGKILL after outbox claim: expired lease is reclaimed once and stale owner is rejected

    Observed result: PASS

    196.4 ms

  5. 05

    FCM outage: backoff prevents immediate retry and provider recovery records one delivery

    Observed result: PASS

    124.3 ms

  6. 06

    Persistent FCM outage stops at retry limit and dead-letters the job

    Observed result: PASS

    156.2 ms

  7. 07

    SIGKILL after simulated FCM acceptance: retry preserves event identity and unique row

    Observed result: PASS

    226.8 ms

  8. 08

    Redis unavailable: OTP checks reject; recovered store retains one-time verification

    Observed result: PASS

    114.3 ms

Durations are test execution times, not endpoint latency measurements.

What was actually exercised.

The limits matter too.

These tests did not stop database daemons, reboot a host or contact live FCM. They did not test the complete production supervisor loop.

Fixtures copied tables, checks and indexes. This M8.1 harness did not copy foreign keys or triggers and did not replay all migrations. It is not the deferred M8.4 clean-database suite.

Notification delivery remains at least once. If a provider accepts a send before the worker crashes, retry can repeat the external send. Stable event identity and one database row do not prove exactly-once delivery to a handset.

See the architecture and order states