The panel built the recovery design, found the structural hole in its own evidence ladder, patched it in-session — and preserved a genuine 2-v-1 on what the 12-million-user re-run still requires.
A recovery design for a managed background-job platform after a poison-pill object (corrupt gzip) was retried 6,842 times in 74 minutes: the crash-reaper path enforced no max_attempts, autoscaling amplified the storm, and the customer dashboard showed "80% complete" throughout. Five candidate mitigations — durable pre-attempt rows, input fingerprinting, a queue circuit breaker, per-shard UI transparency, crash-path retry policy — with a 12M-user re-run due Friday under a 4-hour contractual window.
Every route to another execution — lease expiry, process death, redelivery, manual retry — must pass one retry-decision function with a hard total-execution cap. max_attempts in YAML but not in the reaper is how 6,842 happens.
Incrementing attempt_count only on worker completion means a crash-before-classification leaves no durable marker — durable pre-attempt rows in the lock-acquisition transaction, plus fencing tokens so a stale worker cannot overwrite its replacement.
Two same-class failures with matching length/ETag do not prove the same bytes failed twice. The evidence bar landed on a streaming SHA-256 of bytes already being downloaded: stable identity + byte digest + same deterministic failure twice means bad_input; differing hashes mean unreadable_input; length mismatch means transient_io, never poison.
A worker that crashes mid-decompression never completes a digest — fingerprinting never fires, and the poison object burns the whole attempt cap as lost. Named by all three in the final round: the ladder is a fast-path classifier, not a completeness guarantee. Patched in-session with partial-prefix-hash promotion for caught early aborts.
Draining a multi-GB body just to finish a classification digest can eat the contractual window — abort-on-failure for the patch, drain budgets if ever.
Match a pinned canonical category (code + phase + offset bucket), never message strings — a rolling deploy could reset poison strikes and reopen the storm.
Exclude delayed jobs from depth metrics or cap concurrency — otherwise backoff is undermined by scale-out that adds workers all hitting the same deterministic bad object.
Every path that can create another job execution — including crash recovery — passes one retry-policy function with a hard attempt cap; customer-facing bad_input requires proven byte identity, never metadata.
GO carried 2-1 with the dissent preserved, not smoothed: the majority ships with light on-call and cap alerts; the minority's release conditions are on the record as the price of skipping them.
Grok + Muse: GO with light on-call — storm prevention does not depend on fingerprinting firing; the worst case is bounded unreadable_input, not an infinite loop.
Sol: NO-GO without a canary on representative multi-GB objects, a verified 4-hour capacity budget, and an SRE with a kill switch — 'ship today' is not a safety requirement.