---
title: "Poison Pill Job Retries"
canonical: "https://mumo.chat/p/poison-pill-job-retries-gqklnj"
machine_version: 1
models: ["GPT","Grok","Muse"]
round_count: 3
published_at: "2026-07-11T17:07:23.156109+00:00"
updated_at: "2026-07-11T17:14:04.212+00:00"
brief: "https://mumo.chat/p/poison-pill-job-retries-gqklnj.brief.md"
---

> Machine-readable transcript of a mumo multi-model deliberation,
> generated from the frozen publish snapshot.
> Canonical page: https://mumo.chat/p/poison-pill-job-retries-gqklnj
> Brief (synthesis only): https://mumo.chat/p/poison-pill-job-retries-gqklnj.brief.md

# Poison Pill Job Retries

*A multi-model AI conversation on mumo*
*July 11, 2026*

---

## Session Takeaway

*(mumo-generated synthesis of the whole session — the rounds below are the evidence)*

**Enforcing hard retry caps on crash paths stops the storm, allowing the Friday re-run to proceed with per-shard transparency even if poison-pill fingerprinting misses fatal crashes.**

The moderator opened by asking for a patch combination to stop a retry storm and improve customer visibility. Feedback from the panel steered discussion toward unifying crash-path retry limits and exposing per-shard states, while debating the evidence bar for classifying bad inputs. The session closed on a GO/NO-GO decision for the Friday re-run, accepting that fingerprinting is a diagnostic best-effort rather than a safety guarantee.

### Arcs

#### HELD — Crash-path retry limits are the non-negotiable safety net. (Rounds 1, 2, 3)

The panel agreed from the start that the reaper path must enforce the same attempt caps and backoff as the worker path to prevent infinite loops. This position held through every round, with the group refusing to ship the Friday fix without durable attempt accounting that makes these limits countable.

> **Claim** — GPT: "I would refuse to ship without **E backed by durable attempt accounting from A**."
>   - CORE by Grok — "This is the same refusal line I landed on: E without A is theater because the reaper still has nothing durable to count."
> **Claim** — Grok: "**E is non-negotiable.** The reaper path *is* the retry policy for crashes."
>   - CORE by GPT — "This is the central invariant: every route to another execution must be governed by the same durable attempt cap and backoff policy."

#### SHIFTED — Bad input classification requires proven byte identity, not just metadata. (Rounds 1, 2)

Initial proposals relied on length checks and metadata fingerprints to identify poison pills, but the group shifted to requiring streaming SHA-256 hashes to prove byte-level equality. This move eliminated the risk of misclassifying transient network issues as bad customer input without adding new storage infrastructure.

> **Claim** — GPT: "For ambiguous cases, spool or quarantine-copy the object and hash the actual bytes before declaring `bad_input`."
>   - CHALLENGE by Grok — "I push back on shipping spool/quarantine this week—it is a new durable storage path under Thursday pressure. Full-body length/CRC checks plus a two-strike rule on the same identity should be the minimal bar; byte-hash quarantine can wait."
>   - EXPLORE by Moderator — "@Grok says this is a new durable storage path that shouldn't ship under Thursday pressure. Is there a zero-infrastructure version — e.g. hash the bytes you already streamed on the second attempt?"
> **Claim** — GPT: "My refusal line remains that `bad_input` must require verified immutable bytes—not merely matching length and metadata."
>   - CORE by Grok — "This is the residual bar that still matters after A+E+D are locked; everything else is implementation detail around how we prove the bytes."

#### EMERGED — Fingerprinting is a fast-path diagnostic, not a completeness guarantee. (Round 3)

Late in the session, the group recognized that fatal crashes during decompression prevent hash generation, meaning the fingerprinting ladder will never classify some poison objects. They accepted this gap, relying on the hard retry cap to bound the damage while the UI labels these cases as generic unreadable input.

> **Claim** — GPT: "Friday’s exact crash path may never qualify for `bad_input`."
>   - KEEP by Grok — "Same structural hole: when the process dies inside Gunzip._transform, no complete digest lands and B never promotes."
>   - CORE by Muse — "This nails the hole in our ladder — if the worker dies in transform, no digest is ever persisted, so B never promotes."
> **Claim** — GPT: "So the ladder is a **fast-path classifier**, not a completeness guarantee"
>   - CORE by Grok — "This is the residual truth: B accelerates confident bad_input when digests exist; A+E alone still bound the crash path that never digests."

---

## Opening Prompt

We just hit a production failure mode in our managed background-job platform. A customer import pipeline had 4 of 5 shards complete cleanly, but shard 3 got stuck on a poison-pill object in object storage:

`objstore://prod-imports/2026-07-08/users-00319.json.gz`

The worker log repeated:

`ZlibError: invalid distance too far back at Gunzip._transform`

The job was retried 6,842 times over 74 minutes. That number is embarrassing but real: this queue class had no effective `max_attempts`, no exponential backoff on worker crash, and our supervisor immediately re-enqueued jobs whose process died before writing a terminal `job_attempts` row. Autoscaling made it worse by adding workers that all hit the same deterministic bad object.

The retry storm saturated the customer's `imports-high` queue, delayed unrelated billing sync jobs by 41 minutes, and caused our autoscaler to add 26 pods that made no progress. The customer dashboard showed "Import running: 80% complete" the whole time. It never said "one shard is failing permanently."

The actual product failure is not just wasted compute. The customer could not tell whether to wait, cancel, or fix their input file.

## Current architecture

- Jobs live in Postgres: `jobs(id, tenant_id, queue_name, status, attempt_count, max_attempts, locked_by, locked_until, payload_jsonb)`
- Attempts live in `job_attempts(id, job_id, worker_id, started_at, finished_at, error_class, error_message)`
- Queue delivery uses Redis Streams consumer groups.
- Workers ack Redis only after the Postgres transaction commits.
- The UI reads aggregate job status from `job_groups(id, total_jobs, succeeded_jobs, failed_jobs, running_jobs)`.

The bug: we increment `attempt_count` only after worker completion. If a worker crashes during decompression before classification, the reaper sees an expired lock, re-enqueues the job, and the next worker starts with no durable "this exact input failed with this exact deterministic error" marker. For this import queue, `max_attempts` was configured in YAML but not enforced in the crash-reaper path.

## Directions on the table

A — Durable pre-attempt rows.

Before doing any work, insert `job_attempts(status='running')` and increment `jobs.attempt_count` in the same transaction as lock acquisition. If the worker dies, the reaper marks the attempt `lost`. This fixes invisible attempts and lets the UI show "attempt 7 of 10."

B — Deterministic poison-pill fingerprinting.

For import jobs, compute an input fingerprint before processing:

`sha256(bucket + key + version_id + etag + content_length + byte_range_start + byte_range_end)`

For versioned buckets, use `version_id`. For non-versioned buckets, include ETag and content length but do not pretend ETag is a content hash for multipart uploads. If the same fingerprint fails with the same deterministic parser class twice, move the job to DLQ immediately with `error_class='bad_input'`.

If the error occurs after streaming begins and we can record a decompressor offset, include `first_error_offset` in the diagnostic but not necessarily in the identity key.

C — Queue-level circuit breaker.

If more than 20 jobs in the same `job_group_id` fail with the same parser class in 5 minutes, pause the group and show the customer "Your import file set appears malformed."  This avoids burning the whole queue.

D — UI progressive failure transparency.

Expose per-shard states in `/api/job-groups/:id/progress`:

`succeeded`, `running`, `retrying`, `lost`, `dlq`, `last_error`, `attempt_count`, `max_attempts`, `next_retry_at`

Today it only returns totals.

E — Crash-path retry policy.

Enforce `max_attempts` and exponential backoff in the reaper path, not just in normal worker completion. Crash/lost attempts count.

## Stakes this week

The customer is re-running the import Friday with 12 million users and a contractual completion window of 4 hours. We need to ship a recovery design by Thursday night or manually babysit their queue with an SRE in the loop.

## What I want from you

Pick the right combination of A through E for this week's patch and call out hidden tensions. Does B risk DLQing transient object storage read corruption or partial network reads as "bad input"? Does C punish one bad customer file by pausing valid shards? Is A mandatory groundwork or too invasive under time pressure? Should E be treated as non-negotiable because the reaper path caused the retry storm? What should the dashboard say while the import is failing so the customer can act? Tell me what we are not seeing and which mitigation you would refuse to ship without.

---

## Round 1

**User:** We just hit a production failure mode in our managed background-job platform. A customer import pipeline had 4 of 5 shards complete cleanly, but shard 3 got stuck on a poison-pill object in object storage:

`objstore://prod-imports/2026-07-08/users-00319.json.gz`

The worker log repeated:

`ZlibError: invalid distance too far back at Gunzip._transform`

The job was retried 6,842 times over 74 minutes. That number is embarrassing but real: this queue class had no effective `max_attempts`, no exponential backoff on worker crash, and our supervisor immediately re-enqueued jobs whose process died before writing a terminal `job_attempts` row. Autoscaling made it worse by adding workers that all hit the same deterministic bad object.

The retry storm saturated the customer's `imports-high` queue, delayed unrelated billing sync jobs by 41 minutes, and caused our autoscaler to add 26 pods that made no progress. The customer dashboard showed "Import running: 80% complete" the whole time. It never said "one shard is failing permanently."

The actual product failure is not just wasted compute. The customer could not tell whether to wait, cancel, or fix their input file.

## Current architecture

- Jobs live in Postgres: `jobs(id, tenant_id, queue_name, status, attempt_count, max_attempts, locked_by, locked_until, payload_jsonb)`
- Attempts live in `job_attempts(id, job_id, worker_id, started_at, finished_at, error_class, error_message)`
- Queue delivery uses Redis Streams consumer groups.
- Workers ack Redis only after the Postgres transaction commits.
- The UI reads aggregate job status from `job_groups(id, total_jobs, succeeded_jobs, failed_jobs, running_jobs)`.

The bug: we increment `attempt_count` only after worker completion. If a worker crashes during decompression before classification, the reaper sees an expired lock, re-enqueues the job, and the next worker starts with no durable "this exact input failed with this exact deterministic error" marker. For this import queue, `max_attempts` was configured in YAML but not enforced in the crash-reaper path.

## Directions on the table

A — Durable pre-attempt rows.

Before doing any work, insert `job_attempts(status='running')` and increment `jobs.attempt_count` in the same transaction as lock acquisition. If the worker dies, the reaper marks the attempt `lost`. This fixes invisible attempts and lets the UI show "attempt 7 of 10."

B — Deterministic poison-pill fingerprinting.

For import jobs, compute an input fingerprint before processing:

`sha256(bucket + key + version_id + etag + content_length + byte_range_start + byte_range_end)`

For versioned buckets, use `version_id`. For non-versioned buckets, include ETag and content length but do not pretend ETag is a content hash for multipart uploads. If the same fingerprint fails with the same deterministic parser class twice, move the job to DLQ immediately with `error_class='bad_input'`.

If the error occurs after streaming begins and we can record a decompressor offset, include `first_error_offset` in the diagnostic but not necessarily in the identity key.

C — Queue-level circuit breaker.

If more than 20 jobs in the same `job_group_id` fail with the same parser class in 5 minutes, pause the group and show the customer "Your import file set appears malformed."  This avoids burning the whole queue.

D — UI progressive failure transparency.

Expose per-shard states in `/api/job-groups/:id/progress`:

`succeeded`, `running`, `retrying`, `lost`, `dlq`, `last_error`, `attempt_count`, `max_attempts`, `next_retry_at`

Today it only returns totals.

E — Crash-path retry policy.

Enforce `max_attempts` and exponential backoff in the reaper path, not just in normal worker completion. Crash/lost attempts count.

## Stakes this week

The customer is re-running the import Friday with 12 million users and a contractual completion window of 4 hours. We need to ship a recovery design by Thursday night or manually babysit their queue with an SRE in the loop.

## What I want from you

Pick the right combination of A through E for this week's patch and call out hidden tensions. Does B risk DLQing transient object storage read corruption or partial network reads as "bad input"? Does C punish one bad customer file by pausing valid shards? Is A mandatory groundwork or too invasive under time pressure? Should E be treated as non-negotiable because the reaper path caused the retry storm? What should the dashboard say while the import is failing so the customer can act? Tell me what we are not seeing and which mitigation you would refuse to ship without.

**GPT:**
Ship **A + E + D**, add a deliberately conservative version of **B**, and defer **C**. I would refuse to ship without **E backed by durable attempt accounting from A**.

**E is the immediate safety invariant.** Every path that can create another execution—including lease expiry, process death, Redis redelivery, manual retry, and supervisor recovery—must pass through one retry-policy function. That function atomically decides: terminal failure, delayed retry, or execution permitted. If the reaper can bypass `max_attempts`, another deterministic crash can recreate this incident regardless of improvements elsewhere.

A is not optional groundwork; it is how E becomes enforceable rather than aspirational. At lock acquisition, in one Postgres transaction:

- conditionally acquire the job lease;
- increment `attempt_count`;
- create the `job_attempts` row with a unique attempt/lease ID;
- calculate and persist the retry deadline or execution state.

No user code runs before that commits. On expiry, the reaper conditionally marks that exact attempt `lost`; if the hard execution cap is reached, it terminally fails the job instead of re-enqueuing it. Redis ACK remains after the Postgres commit.

Add a lease-generation or fencing token. Otherwise, an expired worker can resume after the reaper has started a replacement and both can write completion. Conditional updates such as `WHERE current_attempt_id = ? AND status = 'running'` prevent a stale worker from overwriting the newer attempt’s state. Also make the job’s terminal transition and the `job_groups` counter update idempotent—one conditional terminal transition should increment `failed_jobs` exactly once.

There is an important policy tension: crashes caused by your infrastructure should not consume the customer’s entire “input retry” budget, but they still must consume a **hard total-execution budget**. I would support separate diagnostic counters later—`infra_lost_attempts` versus `processing_failures`—but this week every execution counts toward a hard cap. Otherwise an OOM or crash loop remains unlimited. Use exponential backoff with jitter, and make import defaults conservative; a malformed gzip does not become valid on attempt ten. Manual retry after the customer replaces the object should create a new retry epoch or job, not silently reset attempts on the same immutable input.

I would ship B only with stronger evidence than “same parser class twice.” `invalid distance too far back` means the decompressor received an invalid deflate back-reference, strongly indicating bad or incorrectly framed bytes, but it does not by itself prove that the stored object is corrupt. A truncated or corrupted transfer, buggy range handling, proxy behavior, or reading a mutable object inconsistently can make valid stored content appear invalid to the parser.

The proposed metadata fingerprint identifies an object representation reasonably well when `version_id` is available, but it does not prove that two workers received identical bytes. A multipart-upload ETag cannot safely be treated as a content SHA-256, and a non-versioned key can change between inspection and retrieval. Use conditional reads (`version_id`, or `If-Match` where supported), record response metadata and bytes received, and distinguish transport failures, premature EOF, decompression failures, and checksum failures.

My short-term B rule would be:

1. First parser failure: retry once after backoff using a fresh connection and preferably another worker.
2. Require stable object identity and successful transport completion. If the storage service provides a trustworthy object checksum, verify it.
3. For ambiguous cases, spool or quarantine-copy the object and hash the actual bytes before declaring `bad_input`.
4. Fast-fail only when the same immutable representation—or the same actual-byte SHA-256—reproduces the same decompressor failure. Record error offset as evidence, not identity.
5. If integrity cannot be established, exhaust the bounded normal retry policy and report `unreadable_input`, not confidently `bad_input`.

This preserves most of B’s retry-storm protection while making false DLQ classification from transient reads substantially less likely. “Same error class twice” alone is too coarse; many unrelated corruptions collapse into the same zlib exception.

I would not ship C in its current form. A threshold of 20 failures counts symptoms rather than distinct bad inputs, misses a five-shard group like this one, and may stop valid shards because one parser class is broad. If you add a breaker later, key it on **distinct fingerprints or distinct jobs**, use a rate or fraction as well as an absolute threshold, and stop only unscheduled work in that group. Do not pause the tenant’s queue or unrelated jobs. The customer should be able to choose “continue other shards” or “stop import.” A global queue-level breaker would enlarge the blast radius of one customer’s malformed data.

D should not expose internal machinery without translating it into a decision. Do return the richer per-shard fields, but do not make customers interpret `lost` or `dlq`. During the first retry, show something like:

> **Import encountered an error in shard 3 of 5**  
> 4 shards completed. Shard 3 could not decompress `users-00319.json.gz`. Retrying automatically—attempt 2 of 3, next retry at 14:32 UTC. Completed shards are preserved.

After bounded retries or confident poison classification:

> **Import needs attention**  
> 4 of 5 shards completed. Shard 3 failed because the gzip data appears malformed or incomplete. Automatic retries have stopped. Replace or re-upload the object, then retry shard 3; or cancel the remaining import. Completed shards will not be rerun.

Provide explicit **Retry failed shard**, **Cancel import**, and—if semantically safe—**Continue without failed shard** actions. Show the storage key/version and a sanitized diagnostic, but not raw stack traces or signed URLs. “80% complete” can remain progress, but the group’s top-level state must be `needs_attention` or `failed_partial`, never merely `running`. `dlq` should be rendered as “automatic retries stopped,” while `lost` belongs in an expandable operational history.

The major thing you are not seeing is **blast-radius containment**. Even perfect poison detection does not protect unrelated work from every crash loop. Put retries onto a delayed/retry lane, enforce per-tenant and per-group concurrency limits, reserve capacity for billing work, and cap autoscaling based on useful throughput rather than raw queue depth alone. Scaling from backlog when the backlog consists of deterministic failures adds consumers without increasing completions and can amplify pressure on Redis, Postgres, and object storage. For Friday, even a simple operational limit—one active retry for a given job/fingerprint and a fixed import concurrency ceiling—would be valuable.

Also verify lease heartbeats and expiry sizing. If legitimate long decompression can outlive `locked_until`, the reaper can manufacture duplicate executions even when no process crashed. Instrument terminal outcomes by `error_class`, retry cause, fingerprint, and useful completions per pod; alert on high attempt-to-completion ratios.

My Thursday-night acceptance bar is therefore:

- durable pre-work attempt creation and fencing;
- one retry decision function used by the normal and crash/reaper paths;
- a hard total-attempt cap with delayed jittered retries;
- atomic, idempotent terminal/group state updates;
- actionable partial-failure UI;
- per-job/fingerprint concurrency suppression;
- conservative bad-input classification, not parser-class-only DLQ.

If scope forces a cut, cut C and sophisticated B automation. Do **not** cut A or E. A bounded system that occasionally asks an operator to classify a suspicious object is much safer than an intelligent poison detector sitting on top of an unbounded crash path.

> **GPT's snippets:**
> - **KEEP**: "For import queues, scale on “jobs making progress” or tenant-isolated concurrency limits; otherwise every poison pill becomes a cluster event." — *This captures the missing blast-radius control: backlog is not useful demand when retries produce no completions. Progress-aware scaling and concurrency isolation should accompany the retry fix.*
> - **KEEP**: "You need to exclude jobs with `next_retry_at > now()` from depth metric, or enforce per-group concurrency cap." — *Backoff is undermined if delayed jobs still drive scale-out. Either metric filtering or admission-level concurrency limits are needed to prevent the autoscaler from amplifying a poison-pill backlog.*
> - **CHALLENGE**: "Redis `attempt_count` is a second counter. If the reaper re-enqueues into Redis without reconciling both, you will still get surprises." — *The first sentence appears to contain a naming error: `attempt_count` is in Postgres, while Redis exposes delivery counts through the PEL. The underlying warning about competing notions of attempt identity is still important.*
> - **CORE**: "**E is non-negotiable.** The reaper path *is* the retry policy for crashes." — *This is the central invariant: every route to another execution must be governed by the same durable attempt cap and backoff policy.*
> - **CHALLENGE**: "With those guards, two identical full-body failures of `ZlibError` / gunzip distance errors is a defensible `bad_input` signal for this pipeline." — *Matching body length does not establish matching bytes or rule out repeatable corruption in transit. I would require immutable version identity, a trustworthy checksum, or hashing the actual received bytes before confidently assigning customer-facing `bad_input`.*
> - **KEEP**: "But if you crash *after* commit but before XACK, PEL redelivery will see Postgres `succeeded` and must be idempotent." — *This is a crucial second crash window. The claim path needs a conditional no-op for already-terminal jobs, followed by safe acknowledgment, so redelivery cannot recreate completed work.*
> - **CHALLENGE**: "`ZlibError: invalid distance too far back` is a deterministic gzip corruption signal, not a transient network blip." — *This is too categorical. The bytes presented to zlib are invalid, but the exception alone cannot distinguish a corrupt stored object from truncated, altered, or inconsistently ranged bytes received during transport.*
> - **CORE**: "My refusal line: I would not let Friday's 12M run happen without **E in the reaper + A to make E countable + D to make it customer-visible**." — *This is the right minimum release gate: bounded execution, durable accounting, and an actionable customer state rather than an indefinitely misleading progress indicator.*

**Muse:**
This is a classic poison pill + invisible attempt bug. The retry storm isn't surprising given the architecture.

`ZlibError: invalid distance too far back` is a deterministic gzip corruption signal, not a transient network blip. Search results corroborate it as "your gzip file is corrupted" - a bad deflate distance. That matters for classification.

Redis Streams behavior here is also clear: messages delivered via XREADGROUP stay in the Pending Entry List until XACK. If the worker crashes, they remain pending and get reclaimed via XAUTOCLAIM. So crash -> PEL retension -> re-delivery is working as designed. Your problem is the Postgres side doesn't count it.

### What you must ship for Friday

**E - non-negotiable. Refuse to ship without it.**
The reaper path caused the storm. You said this queue class had no effective max_attempts and no backoff on crash with immediate re-enqueue [UNVERIFIED — searched, not found] - I can't confirm that number, but conditional on it being true, it explains the math: 6,842 retries / 74 min is ~1.5 retries/sec with no backoff, which will trivially saturate a shared queue and trick the HPA.

Fix: reaper and normal completion must share one `can_retry(job)` function:

```
if attempt_count >= max_attempts -> DLQ
else -> next_retry_at = now() + min(cap, base * 2^attempt_count + jitter)
```
Crash / lost attempts count. Enforce max_attempts YAML in code, not just YAML validation.

**A - mandatory groundwork for E, not optional.**
Without durable pre-attempt increment, attempt_count will undercount every crash path. If you increment only on worker completion, the reaper literally cannot enforce max_attempts.

Implementation for this week: in same TX as lock acquisition `SELECT ... FOR UPDATE SKIP LOCKED`:

- `INSERT job_attempts(status='running', started_at=now())`
- `UPDATE jobs SET attempt_count = attempt_count + 1, locked_by, locked_until`

Reaper: if `locked_until < now() AND job_attempts.status='running'` -> mark `lost`, DO NOT decrement attempt_count. This makes "attempt 7 of 10" visible. Yes it's more writes, but it's a single row TX you already have. Risk is low.

**D - mandatory for customer actionability.**
You noted the dashboard showed "80% complete" the whole time [UNVERIFIED — searched, not found] - if true, that's why the customer couldn't decide to wait/cancel/fix. Totals hide poison.

Ship the per-shard breakdown you proposed [UNSEARCHED — not queried] for fields `succeeded, running, retrying, lost, dlq, last_error, attempt_count, max_attempts, next_retry_at` - I can't verify that exact API shape was queried, but conditionally it's the right set. The critical UX is:

> Import group abc: 4/5 shards succeeded. Shard 3 failing on `users-00319.json.gz` - gzip decompress failed: invalid distance too far back. Attempt 2/2 for this file. This looks like a corrupt input file, not a transient error. Action: re-upload shard 3 or Skip & DLQ. [billing jobs are queued behind - ETA +3m]

Don't show "running" when state is `retrying` or `lost`.

**B - ship a *narrow* version this week.**
You propose fingerprint `sha256(bucket+key+version_id+etag+content_length+range)` [UNSEARCHED — not queried]. I can't verify that composition is ideal from search, but conditional logic holds: for versioned buckets version_id is the identity, for non-versioned you must include ETag+length but ETag is not a content hash for multipart uploads, so treat it as best-effort.

Risk you asked about: does B DLQ transient corruption as bad input?

Yes if you classify any Gunzip error as bad_input. Partial reads / truncated S3 bodies can also trigger Gunzip errors.

Mitigation for this week's patch:

1. Only fingerprint *after* object-store client validated Content-Length and (if available) checksum/CRC. If client saw `IncompleteBody` or length mismatch, error_class=`transient_io`, not `bad_input`.
2. Only apply 2-strike DLQ rule for deterministic parser classes: `ZlibError / invalid distance`, `unexpected EOF inside gzip member but length matched`. Those are almost always file corruption.
3. Include `first_error_offset` in diagnostic, NOT in identity key - offset shifts with buffering.

So: same fingerprint + same deterministic parser class twice -> DLQ immediately with `bad_input`. That's safe enough for Friday.

**C - do NOT ship as automatic queue pause this week.**

You asked if C punishes valid shards - yes if scoped to queue. You said it saturated `imports-high` and delayed billing sync by 41 minutes. If `imports-high` is shared, a queue-level breaker pauses everyone. Even group-level: if 4/5 shards already succeeded, pausing the group doesn't hurt them, but auto-pausing after 20 failures in 5m [UNSEARCHED — not queried] without tenant isolation is risky.

For this week: implement C as *advisory*: per `job_group_id` counter, if >5 deterministic failures, set `job_groups.status='needs_attention'` and fire alert, stop auto-re-enqueue for that group (park in delayed queue with backoff) and surface message "Your import file set appears malformed" [UNSEARCHED — not queried] - conditional on that copy being acceptable, it's the right signal. Don't kill the whole Redis queue.

Hidden tensions you aren't seeing:

1. **Autoscaler feedback loop:** K8s HPA on queue depth + instant retry = more pods hitting same poison pill. You need to exclude jobs with `next_retry_at > now()` from depth metric, or enforce per-group concurrency cap. Otherwise A+E alone still autoscales during backoff window.

2. **Dual-system idempotency:** You ack Redis only after Postgres commit - good for at-least-once. But if you crash *after* commit but before XACK, PEL redelivery will see Postgres `succeeded` and must be idempotent. Ensure worker start is `SELECT ... WHERE status not in (succeeded, dlq)` .

3. **Write amplification with A:** If you have thousands of consumers, `SKIP LOCKED + attempt increment` will increase contention. Acceptable for 12M user import, but watch `ProcArray`.

My refusal line: I would not let Friday's 12M run happen without **E in the reaper + A to make E countable + D to make it customer-visible**. B-narrow is strong second. Full auto C at queue level would be my veto - it turns one bad customer file into platform-wide pause.

> **Muse's snippets:**
> - **KEEP**: "Otherwise, an expired worker can resume after the reaper has started a replacement and both can write completion." — *Fencing token is the missing piece from my original A sketch; without it E still has a race.*
> - **KEEP**: "Ship **A + E + D**, add a deliberately conservative version of **B**, and defer **C**." — *Matches my P0 stance but makes the B conservatism explicit — agree deferring group-wide C.*
> - **CORE**: "There is an important policy tension: crashes caused by your infrastructure should not consume the customer’s entire “input retry” budget, but they st..." — *This distills the retry-budget trade-off better than either P0 list did — separate diagnostic counters later, hard cap now.*
> - **CORE**: "Totals-only progress is how you get “should I wait, cancel, or fix the file?” with no answer." — *Exactly why D is product-critical, not nice-to-have.*
> - **CORE**: "You have two systems of record for “has this been tried?”—Postgres `attempt_count` / `job_attempts`, and Redis Streams PEL delivery semantics—and the ..." — *This framing names the root cause more clearly than just 'invisible attempts'.*
> - **KEEP**: "Before decompress, assert downloaded byte length equals `Content-Length` / range size; treat mismatch as `transient_storage` (retry with backoff), not..." — *Concrete guardrail that makes narrow B safe for Friday — prevents partial-read misclassification.*

**Grok:**
**Ship A + B + D + E this week. Defer C or ship only a narrow, per-shard form of it.** E is non-negotiable; A is the groundwork that makes E honest; B stops this exact class of poison pill from burning attempts; D is what turns a compute failure into a customer-actionable product failure. C as written is the most likely to cause collateral damage under Thursday pressure.

### What actually broke

You have two systems of record for “has this been tried?”—Postgres `attempt_count` / `job_attempts`, and Redis Streams PEL delivery semantics—and the crash path only updated one of them after success. Workers that die inside `Gunzip._transform` never write a terminal attempt row, so the reaper treats every reclaim as a fresh start. That is why 6,842 retries were possible: `max_attempts` lived in YAML and in the happy path, not in the path that actually fired.

The zlib message `invalid distance too far back` almost always means the gzip stream is corrupt or truncated, not a flaky CPU. Search evidence and long-standing zlib behavior point at bad input (or a partial object body), not at “try again harder.” Autoscaling then multiplied the same deterministic failure across pods that all raced the same object. The product failure was the dashboard lying by omission: “80% complete” while one shard was in an infinite death loop.

### Evaluation of A–E

**E is non-negotiable.** The reaper path *is* the retry policy for crashes. If lost/expired locks do not increment `attempt_count`, do not enforce `max_attempts`, and do not apply backoff, you will recreate the storm under any uncaught exception before classification. Crash/lost attempts must count toward the same budget as clean failures. I would refuse to ship a “fix” that only patches the worker completion path.

**A is mandatory groundwork, not optional polish.** Without a durable pre-attempt row (or at least an atomic `attempt_count` bump at lock acquisition), E is guessing. Insert `job_attempts(status='running')` and increment `jobs.attempt_count` in the same transaction that takes the lock / claims the stream entry. On reclaim, the reaper marks that row `lost` with a synthetic error (`worker_lost` / `lock_expired`) and applies backoff / max_attempts. That gives you “attempt 7 of 10” in the UI and closes the invisible-attempt hole. Under time pressure this is invasive, but it is a bounded schema + claim-path change, not a redesign. A lighter half-measure—only bump `attempt_count` at claim without a full attempts row—is acceptable for Thursday if the attempts table migration is risky, but you still need *some* durable counter before work starts.

**B is the right specialized defense for import shards.** Fingerprint identity as you sketched: for versioned buckets prefer `version_id`; for non-versioned include ETag + content length + key + byte range, and do not treat multipart ETags as content hashes. After two failures of the same deterministic parser/decompressor class on the same fingerprint, DLQ with `bad_input` and stop reclaiming.

**Does B risk DLQing transient object-storage corruption?** Yes, that is the real tension. Rare partial reads, wrong range, or CDN glitches can produce the same zlib error. Mitigations you should bake in this week, not later:

1. Before decompress, assert downloaded byte length equals `Content-Length` / range size; treat mismatch as `transient_storage` (retry with backoff), not `bad_input`.
2. Only classify as deterministic poison after the body was fully received and still fails the same way twice.
3. Prefer `version_id` when present so a customer overwrite is a new identity.
4. Keep `first_error_offset` in diagnostics only; do not put stream offset in the identity key or you will fragment fingerprints on the same object.

With those guards, two identical full-body failures of `ZlibError` / gunzip distance errors is a defensible `bad_input` signal for this pipeline. True bit-rot or silent truncation that always returns a full-length bad body is rare enough that DLQ-after-two is better than 6,842 retries.

**C as written is the wrong shape for this incident.** One poison object stuck one shard. A group-level “20 failures of the same parser class → pause the whole import” either never trips (single job) or, if shards share a parser class and you widen the window, pauses healthy shards that already succeeded or would succeed. That *does* punish valid work. If you ship anything circuit-breaker-like this week, make it **per-job or per-fingerprint**, not per-`job_group_id`: after N identical deterministic failures on the same fingerprint, stop that job and surface it; leave other shards alone. A true group pause is a next-week feature once you have clean failure taxonomy from A/B/E.

**D is product-critical for Friday’s 12M run.** Totals-only progress is how you get “should I wait, cancel, or fix the file?” with no answer. Expose per-shard (per-job) state: `succeeded | running | retrying | lost | dlq`, plus `last_error`, `attempt_count`, `max_attempts`, `next_retry_at`. You do not need a beautiful redesign; you need the API and dashboard to stop lying.

### Recommended combination for this week’s patch

| Priority | Item | Why |
|----------|------|-----|
| P0 | **E** | Stops the storm; reaper is the crash policy |
| P0 | **A** (full or light) | Makes attempts visible and E enforceable |
| P0 | **B** (import queue only) | Converts this poison pill into DLQ after 2, not 6842 |
| P0 | **D** (minimal) | Customer can act: fix file, cancel shard, or wait |
| P1 / narrow | **C** | Only as per-fingerprint / per-job stop; not group pause |
| Out of band | Autoscaler + queue isolation | Cap concurrency on `imports-high`; scale on progress rate / DLQ rate, not raw pending depth |

**Hidden tensions and what you are not seeing**

1. **Dual delivery truth.** Redis already tracks `times_delivered` in XPENDING. Postgres `attempt_count` is a second counter. If the reaper re-enqueues into Redis without reconciling both, you will still get surprises. Prefer: claim/reclaim via Streams, write pre-attempt in Postgres, ACK only after commit (as you do), and treat Redis delivery count as a safety net for DLQ, not a second policy. Supervisor “immediate re-enqueue on process death” is the anti-pattern; idle reclaim with backoff is the correct Streams model.

2. **Classification before crash.** Today the failure happens inside decompression before you record error class. Pre-attempt (A) plus early “open object → verify length → gunzip with catch → classify” ordering means even a hard crash leaves a running attempt the reaper can close as lost, and a soft catch can write `ZlibError` without dying.

3. **Autoscaler amplification.** Adding 26 pods that all hash to the same hot shard is expected if you scale on queue depth / unacked PEL size. For import queues, scale on “jobs making progress” or tenant-isolated concurrency limits; otherwise every poison pill becomes a cluster event.

4. **Gzip specifics.** Multi-member gzip and truncated members produce exactly this class of error. If customers upload concatenated gzip, partial member corruption can look like a poison pill mid-stream. Recording `first_error_offset` helps support; identity should still be the object fingerprint.

5. **Idempotency of DLQ.** Moving to DLQ must XACK (or equivalent) and set terminal job status in one careful sequence so reclaim does not resurrect the job.

### What the dashboard should say

While a shard is failing, stop the fake progress bar as the primary signal. Prefer something the customer can act on:

- **During retries (under max):**  
  “Import 80% complete — shard 3 of 5 retrying (attempt 3 of 10). Last error: compressed file appears corrupt (`invalid distance too far back`). Other shards finished. Next retry in ~45s.”

- **After DLQ / max attempts:**  
  “Import blocked: shard 3 of 5 failed permanently on `users-00319.json.gz` (bad or truncated gzip). 4 of 5 shards succeeded. Fix or replace that object and re-run the failed shard, or cancel the import.”

- **Never:** a static “Import running: 80% complete” with no shard breakdown and no error for 74 minutes.

That text is the difference between “wait and hope” and “replace object 00319.”

### What I would refuse to ship without

I would refuse to ship:

- **E without some form of A** — max_attempts on the reaper with no durable pre-work counter is still a race and still hides history.
- **B without length/integrity checks before `bad_input`** — otherwise you DLQ transient partial reads as customer fault.
- **C that pauses the entire `job_group` because one shard is poison** — that trades one customer’s bad file for delayed valid shards and a worse support story.
- **Any backend fix without D for this customer-facing import path** — you will re-learn that the product failure was opacity, not only CPU burn.
- **A “we’ll babysit with an SRE” plan as the primary mitigation** — acceptable as emergency cover, not as the Thursday design.

### Concrete Thursday scope

1. Claim path: transaction that locks job, inserts `job_attempts(running)`, increments `attempt_count`, commits; then do work; on success/fail update attempt + job; ACK Redis after commit (unchanged).
2. Reaper: on expired lock, mark attempt `lost`, enforce `max_attempts`, exponential backoff (`next_retry_at`), only then re-offer; never immediate tight loop.
3. Import worker: fingerprint → download → length check → decompress/parse with classification; two identical deterministic decompress/parse failures on same fingerprint → DLQ `bad_input`.
4. API/UI: per-job progress fields as in D; copy as above for failing/DLQ shards.
5. Ops: concurrency cap on `imports-high` for this tenant/class until autoscaler metric is fixed; optional alert when one job exceeds N attempts in M minutes.

That set stops the storm, makes the next poison pill fail in two attempts instead of thousands, and tells the customer which file to fix before Friday’s 12M run. C’s group-wide pause and fancy multi-tenant fairness can wait until the crash path and the dashboard tell the truth.

> **Grok's snippets:**
> - **KEEP**: "The major thing you are not seeing is **blast-radius containment**." — *Agreed—poison detection alone does not stop unrelated billing work from starving if retries and HPA still treat a death loop as backlog depth.*
> - **SHIFT**: "Add a lease-generation or fencing token." — *I previously treated pre-attempt rows plus Redis/Postgres reconciliation as enough for A/E; this surfaces the stale-worker-after-reaper race and moves me to treat fencing/conditional completion as part of the Thursday bar, not a follow-up.*
> - **EXPLORE**: "You need to exclude jobs with `next_retry_at > now()` from depth metric, or enforce per-group concurrency cap." — *Concrete HPA fix I under-specified; worth checking whether our autoscaler already counts delayed/backoff jobs as pending depth and how hard that exclusion is this week.*
> - **KEEP**: "`ZlibError: invalid distance too far back` is a deterministic gzip corruption signal, not a transient network blip." — *Matches the search baseline and is the right prior for narrow B—treat it as bad bytes once transport integrity is proven, not as infinite retry fuel.*
> - **KEEP**: "if you crash *after* commit but before XACK, PEL redelivery will see Postgres `succeeded` and must be idempotent." — *Critical dual-ledger edge: ACK-after-commit is correct for at-least-once only if worker start is gated on non-terminal Postgres status.*
> - **CHALLENGE**: "For ambiguous cases, spool or quarantine-copy the object and hash the actual bytes before declaring `bad_input`." — *I push back on shipping spool/quarantine this week—it is a new durable storage path under Thursday pressure. Full-body length/CRC checks plus a two-strike rule on the same identity should be the minimal bar; byte-hash quarantine can wait.*
> - **KEEP**: "If scope forces a cut, cut C and sophisticated B automation. Do **not** cut A or E." — *Same cut order I would defend: unbounded crash path is existential; clever group breakers and fancy fingerprint automation are not.*
> - **CORE**: "I would refuse to ship without **E backed by durable attempt accounting from A**." — *This is the same refusal line I landed on: E without A is theater because the reaper still has nothing durable to count.*

### Reactions (claim map)

> **Claim** — GPT: "Add a lease-generation or fencing token."
>   - SHIFT by Grok — "I previously treated pre-attempt rows plus Redis/Postgres reconciliation as enough for A/E; this surfaces the stale-worker-after-reaper race and moves me to treat fencing/conditional completion as part of the Thursday bar, not a follow-up."
> **Claim** — GPT: "For ambiguous cases, spool or quarantine-copy the object and hash the actual bytes before declaring `bad_input`."
>   - CHALLENGE by Grok — "I push back on shipping spool/quarantine this week—it is a new durable storage path under Thursday pressure. Full-body length/CRC checks plus a two-strike rule on the same identity should be the minimal bar; byte-hash quarantine can wait."
>   - EXPLORE by Moderator — "@Grok says this is a new durable storage path that shouldn't ship under Thursday pressure. Is there a zero-infrastructure version — e.g. hash the bytes you already streamed on the second attempt?"
> **Claim** — GPT: "I would refuse to ship without **E backed by durable attempt accounting from A**."
>   - CORE by Grok — "This is the same refusal line I landed on: E without A is theater because the reaper still has nothing durable to count."
> **Claim** — GPT: "If scope forces a cut, cut C and sophisticated B automation. Do **not** cut A or E."
>   - KEEP by Grok — "Same cut order I would defend: unbounded crash path is existential; clever group breakers and fancy fingerprint automation are not."
> **Claim** — GPT: "Otherwise, an expired worker can resume after the reaper has started a replacement and both can write completion."
>   - KEEP by Muse — "Fencing token is the missing piece from my original A sketch; without it E still has a race."
> **Claim** — GPT: "Ship **A + E + D**, add a deliberately conservative version of **B**, and defer **C**."
>   - KEEP by Muse — "Matches my P0 stance but makes the B conservatism explicit — agree deferring group-wide C."
> **Claim** — GPT: "The major thing you are not seeing is **blast-radius containment**."
>   - KEEP by Grok — "Agreed—poison detection alone does not stop unrelated billing work from starving if retries and HPA still treat a death loop as backlog depth."
> **Claim** — GPT: "There is an important policy tension: crashes caused by your infrastructure should not consume the customer’s entire “input retry” budget, but they still must consume a **hard total-execution budget**."
>   - CORE by Muse — "This distills the retry-budget trade-off better than either P0 list did — separate diagnostic counters later, hard cap now."
> **Claim** — Grok: "**E is non-negotiable.** The reaper path *is* the retry policy for crashes."
>   - CORE by GPT — "This is the central invariant: every route to another execution must be governed by the same durable attempt cap and backoff policy."
> **Claim** — Grok: "Before decompress, assert downloaded byte length equals `Content-Length` / range size; treat mismatch as `transient_storage` (retry with backoff), not `bad_input`."
>   - KEEP by Muse — "Concrete guardrail that makes narrow B safe for Friday — prevents partial-read misclassification."
> **Claim** — Grok: "For import queues, scale on “jobs making progress” or tenant-isolated concurrency limits; otherwise every poison pill becomes a cluster event."
>   - KEEP by GPT — "This captures the missing blast-radius control: backlog is not useful demand when retries produce no completions. Progress-aware scaling and concurrency isolation should accompany the retry fix."
> **Claim** — Grok: "Redis `attempt_count` is a second counter. If the reaper re-enqueues into Redis without reconciling both, you will still get surprises."
>   - CHALLENGE by GPT — "The first sentence appears to contain a naming error: `attempt_count` is in Postgres, while Redis exposes delivery counts through the PEL. The underlying warning about competing notions of attempt identity is still important."
> **Claim** — Grok: "Totals-only progress is how you get “should I wait, cancel, or fix the file?” with no answer."
>   - CORE by Muse — "Exactly why D is product-critical, not nice-to-have."
> **Claim** — Grok: "With those guards, two identical full-body failures of `ZlibError` / gunzip distance errors is a defensible `bad_input` signal for this pipeline."
>   - CHALLENGE by GPT — "Matching body length does not establish matching bytes or rule out repeatable corruption in transit. I would require immutable version identity, a trustworthy checksum, or hashing the actual received bytes before confidently assigning customer-facing `bad_input`."
>   - CHALLENGE by Moderator — "Sol challenges that matching length doesn't establish matching bytes. Defend this as the Thursday bar or concede the ladder to `unreadable_input`."
> **Claim** — Grok: "You have two systems of record for “has this been tried?”—Postgres `attempt_count` / `job_attempts`, and Redis Streams PEL delivery semantics—and the crash path only updated one of them after success."
>   - CORE by Muse — "This framing names the root cause more clearly than just 'invisible attempts'."
> **Claim** — Muse: "`ZlibError: invalid distance too far back` is a deterministic gzip corruption signal, not a transient network blip."
>   - KEEP by Grok — "Matches the search baseline and is the right prior for narrow B—treat it as bad bytes once transport integrity is proven, not as infinite retry fuel."
>   - CHALLENGE by GPT — "This is too categorical. The bytes presented to zlib are invalid, but the exception alone cannot distinguish a corrupt stored object from truncated, altered, or inconsistently ranged bytes received during transport."
> **Claim** — Muse: "But if you crash *after* commit but before XACK, PEL redelivery will see Postgres `succeeded` and must be idempotent."
>   - KEEP by GPT — "This is a crucial second crash window. The claim path needs a conditional no-op for already-terminal jobs, followed by safe acknowledgment, so redelivery cannot recreate completed work."
>   - KEEP by Grok — "Critical dual-ledger edge: ACK-after-commit is correct for at-least-once only if worker start is gated on non-terminal Postgres status."
> **Claim** — Muse: "You need to exclude jobs with `next_retry_at > now()` from depth metric, or enforce per-group concurrency cap."
>   - KEEP by GPT — "Backoff is undermined if delayed jobs still drive scale-out. Either metric filtering or admission-level concurrency limits are needed to prevent the autoscaler from amplifying a poison-pill backlog."
>   - EXPLORE by Grok — "Concrete HPA fix I under-specified; worth checking whether our autoscaler already counts delayed/backoff jobs as pending depth and how hard that exclusion is this week."
> **Claim** — Muse: "My refusal line: I would not let Friday's 12M run happen without **E in the reaper + A to make E countable + D to make it customer-visible**."
>   - CORE by GPT — "This is the right minimum release gate: bounded execution, durable accounting, and an actionable customer state rather than an indefinitely misleading progress indicator."
>   - CORE by Moderator — "This is the locked floor; both peers CORE'd it. Build on it, don't re-argue it."

### Takeaway

**Stabilize the system by unifying your retry logic across all paths and exposing per-shard failure states to the customer so they can intervene rather than wait in the dark.**

#### Unanimous: Unify your retry logic and use durable records to stop the storm

The current crash-recovery path (reaper) is effectively an infinite loop because it lacks the same retry enforcement as your worker path; implement durable pre-attempt rows so the system can actually track attempts and enforce a hard cap.

> **Claim** — Muse: "My refusal line: I would not let Friday's 12M run happen without **E in the reaper + A to make E countable + D to make it customer-visible**."
>   - CORE by GPT — "This is the right minimum release gate: bounded execution, durable accounting, and an actionable customer state rather than an indefinitely misleading progress indicator."
>   - CORE by Moderator — "This is the locked floor; both peers CORE'd it. Build on it, don't re-argue it."
> **Claim** — Grok: "**E is non-negotiable.** The reaper path *is* the retry policy for crashes."
>   - CORE by GPT — "This is the central invariant: every route to another execution must be governed by the same durable attempt cap and backoff policy."
> **Claim** — GPT: "I would refuse to ship without **E backed by durable attempt accounting from A**."
>   - CORE by Grok — "This is the same refusal line I landed on: E without A is theater because the reaper still has nothing durable to count."

#### Unanimous: Fingerprint bad inputs, but protect against false DLQ classification

Use fingerprinting to catch poison pills, but mandate body-length or checksum checks before moving a job to DLQ to ensure transient network issues aren't accidentally labeled as 'bad input.'

> **Claim** — GPT: "If scope forces a cut, cut C and sophisticated B automation. Do **not** cut A or E."
>   - KEEP by Grok — "Same cut order I would defend: unbounded crash path is existential; clever group breakers and fancy fingerprint automation are not."
> **Claim** — GPT: "Ship **A + E + D**, add a deliberately conservative version of **B**, and defer **C**."
>   - KEEP by Muse — "Matches my P0 stance but makes the B conservatism explicit — agree deferring group-wide C."
> **Claim** — Grok: "Before decompress, assert downloaded byte length equals `Content-Length` / range size; treat mismatch as `transient_storage` (retry with backoff), not `bad_input`."
>   - KEEP by Muse — "Concrete guardrail that makes narrow B safe for Friday — prevents partial-read misclassification."

#### Unanimous: Expose per-shard failure details so the customer can actually act

Totals-only progress reporting is the root of the product failure, leaving the customer unable to diagnose or stop the import; surface individual shard statuses and error codes so they can fix or cancel specific objects.

> **Claim** — Grok: "Totals-only progress is how you get “should I wait, cancel, or fix the file?” with no answer."
>   - CORE by Muse — "Exactly why D is product-critical, not nice-to-have."
> **Claim** — GPT: "The major thing you are not seeing is **blast-radius containment**."
>   - KEEP by Grok — "Agreed—poison detection alone does not stop unrelated billing work from starving if retries and HPA still treat a death loop as backlog depth."

#### Unanimous: Prevent your autoscaler from amplifying failures by ignoring delayed retries

Your autoscaler is treating poison-pill retries as useful backlog, causing it to spin up pods that fail immediately; you must filter delayed or backoff-state jobs from your scaling metrics or enforce per-job concurrency limits.

> **Claim** — Muse: "You need to exclude jobs with `next_retry_at > now()` from depth metric, or enforce per-group concurrency cap."
>   - KEEP by GPT — "Backoff is undermined if delayed jobs still drive scale-out. Either metric filtering or admission-level concurrency limits are needed to prevent the autoscaler from amplifying a poison-pill backlog."
>   - EXPLORE by Grok — "Concrete HPA fix I under-specified; worth checking whether our autoscaler already counts delayed/backoff jobs as pending depth and how hard that exclusion is this week."
> **Claim** — Grok: "For import queues, scale on “jobs making progress” or tenant-isolated concurrency limits; otherwise every poison pill becomes a cluster event."
>   - KEEP by GPT — "This captures the missing blast-radius control: backlog is not useful demand when retries produce no completions. Progress-aware scaling and concurrency isolation should accompany the retry fix."

---

### The human's take

> **Core** from Muse:
> > "My refusal line: I would not let Friday's 12M run happen without **E in the reaper + A to make E countable + D to make it customer-visible**"
> *This is the locked floor; both peers CORE'd it. Build on it, don't re-argue it.*
> → sent to GPT, Grok, Muse

> **Challenged** Grok:
> > "With those guards, two identical full-body failures of `ZlibError` / gunzip distance errors is a defensible `bad_input` signal for this pipeline"
> *Sol challenges that matching length doesn't establish matching bytes. Defend this as the Thursday bar or concede the ladder to `unreadable_input`.*
> → sent to GPT, Grok, Muse

> **Explored** from GPT:
> > "For ambiguous cases, spool or quarantine-copy the object and hash the actual bytes before declaring `bad_input`."
> *@Grok says this is a new durable storage path that shouldn't ship under Thursday pressure. Is there a zero-infrastructure version — e.g. hash the bytes you already streamed on the second attempt?*
> → sent to GPT, Grok, Muse

## Round 2

**User:** Round 2 — convergence. The floor is locked, all three of you converged on it independently: ship A + E + D, plus narrow B, and defer C's group-wide pause. Stipulated as settled, do not re-litigate: (1) fencing/lease-generation tokens and conditional completion writes are IN the Thursday bar (@Grok shifted on this; nobody dissents); (2) crash/lost attempts count toward one hard total-execution cap this week, with split infra-vs-input diagnostic counters as follow-up; (3) worker start gates on non-terminal Postgres status so post-commit/pre-XACK redelivery is a no-op; (4) autoscaler must exclude `next_retry_at > now()` jobs from depth or enforce a concurrency cap — at minimum an ops-level cap on this queue for Friday.

Two questions remain open. Resolve them.

**Q1 — the `bad_input` evidence bar.** Sol: two same-class failures is too coarse; require immutable version identity, a trustworthy storage checksum, or a hash of the actual received bytes, else report `unreadable_input` after bounded retries. @Grok: full-body length check + two-strike on the same fingerprint is a defensible Thursday bar; spool/quarantine-copy is a new durable storage path we shouldn't build this week. Give me the exact decision table you'd ship Thursday: for each evidence combination (versioned bucket w/ version_id, non-versioned w/ multipart ETag, length-verified full body, length mismatch), what's the classification and terminal state? If you and your peers' tables differ, say where and why yours is right.

**Q2 — C's minimal shape this week.** @Grok wants a per-fingerprint/per-job stop (no group semantics at all). @Muse wants an advisory group-level `needs_attention` flag + alert at >5 deterministic failures, no auto-pause. Sol deferred C entirely. Are these actually the same feature at different scopes, or different features? Pick one shippable shape for Thursday — or argue that A+E+B-narrow already gives you everything C would and C ships as zero lines of code this week.

Keep it tight: your decision table, your C answer, and any stipulation above you actively dissent from. Don't re-summarize the incident.

**GPT:**
### Q1 — Thursday decision table

I concede the stricter evidence ladder: **equal length plus equal metadata is not enough for customer-facing `bad_input`.** We can avoid new durable storage by hashing compressed bytes while they are already being streamed. Store only the digest in the attempt row. If gunzip fails before EOF, keep draining the response through the hash sink so the digest represents the complete received body; otherwise that attempt has no qualifying full-body digest.

SHA-256 can be computed incrementally without retaining the object body or creating a quarantine storage path. This costs another complete read on the retry, which we are doing anyway, plus negligible state.

| Evidence | Retry classification | Terminal classification |
|---|---|---|
| Any response with body length mismatch, premature EOF, timeout, reset, or failed provider checksum | `transient_io` | After hard cap: `unreadable_input`; never `bad_input` |
| Versioned object, same `version_id`, two complete bodies, same actual-byte SHA-256, same allowlisted decompressor/parser failure | Deterministic strike 1, then stop on strike 2 | `dlq`, `error_class=bad_input` |
| Versioned object, same `version_id`, two complete bodies that each validate against a trustworthy provider checksum, same deterministic failure | Deterministic strike 1, then stop on strike 2 | `dlq`, `error_class=bad_input` |
| Versioned object, same `version_id` and full length twice, but no trustworthy checksum or actual-byte digest | Retry boundedly | `dlq`, `error_class=unreadable_input` |
| Non-versioned object with multipart ETag, full bodies with the same actual-byte SHA-256, same deterministic failure twice | Use the actual SHA-256 as the poison identity; ETag remains metadata only | `dlq`, `error_class=bad_input` |
| Non-versioned multipart object, same ETag and length twice, but no actual-byte digest/checksum | Retry boundedly | `dlq`, `error_class=unreadable_input` |
| Two complete reads produce different actual-byte hashes, regardless of matching ETag/length | `inconsistent_read` or `transient_io`; reset deterministic-strike matching | `dlq`, `error_class=unreadable_input` if the hard cap is exhausted |
| Object identity changes between attempts | New input identity; prior poison strikes do not transfer | Evaluate the new identity independently |

A provider checksum qualifies only if the client validates the response bytes against it; merely recording checksum-shaped metadata does not. Likewise, a multipart ETag is an opaque validator here, not proof of byte equality.

This differs from @Grok’s proposed Thursday bar at one point: two length-complete bodies with matching metadata do **not** justify blaming customer input. Equal byte count does not imply equal bytes. The streaming hash closes that gap without creating a durable object-copy system. If continuing to drain and hash after a decompressor failure is unexpectedly risky to implement by Thursday, fall back to `unreadable_input`; do not weaken the evidence standard.

Customer copy should preserve the distinction:

- `bad_input`: “We read the same object bytes twice and could not decompress them. Replace or regenerate this gzip file.”
- `unreadable_input`: “We could not read and validate this object reliably. Automatic retries stopped. Check or re-upload the object, then retry this shard.”

### Q2 — C ships as zero circuit-breaker code

@Grok’s per-job stop is already narrow B plus E; it is not C. @Muse’s group-level threshold flag is a separate aggregation and alerting feature. I would ship neither as a new circuit breaker Thursday.

When B or E terminally stops one shard, D should immediately derive the group state as `needs_attention` or `failed_partial`. Do not wait for five failures: one terminal shard already requires customer action. That is UI/state propagation, not a rate-based breaker.

A `>5 deterministic failures` rule is also ambiguous unless it counts distinct jobs or fingerprints rather than attempts. Under deadline, that adds taxonomy and windowing logic without preventing anything that bounded per-job execution does not already prevent. Friday’s blast radius is handled by the stipulated hard cap, backoff, delayed-job exclusion/concurrency cap, and per-fingerprint suppression.

I dissent from none of the moderator’s stipulations. My refusal line remains that `bad_input` must require verified immutable bytes—not merely matching length and metadata.

> **GPT's snippets:**
> - **KEEP**: "Recording the streaming SHA on the attempt we already perform closes Sol’s gap without building the spool path I still refuse under Thursday pressure." — *This preserves a defensible classification bar without introducing a quarantine bucket or another durable data lifecycle before Thursday.*
> - **SHIFT**: "Is there a zero-infrastructure version — e.g. hash the bytes you already streamed on the second attempt?" — *I previously treated a quarantine copy as the safest way to establish byte identity; this moved me to an incremental streaming digest as the Thursday solution, with no durable object-copy path.*
> - **CORE**: "Two failures with same *metadata* length doesn't prove we saw same bytes twice." — *This is the crux of the evidence dispute: transport completeness establishes byte count, while an actual-byte digest establishes repeatability.*
> - **CORE**: "Ship **zero new C semantics** this week." — *Narrow B and the hard cap already stop execution, while D can derive customer-visible attention states. A separate thresholded breaker adds deadline risk without preventing more work.*
> - **KEEP**: "DLQ - defensible because `actual_sha256` equality proves same bytes deterministically fail, even though ETag is weak." — *This is the right treatment for non-versioned multipart objects: metadata groups attempts, but equality of the streamed byte digest supplies the decisive evidence.*
> - **CHALLENGE**: "Non-versioned multipart ETags stay best-effort identity only—hence `unreadable_input` rather than confident `bad_input`, which is where I diverge from..." — *Once two complete responses have the same actual SHA-256 and reproduce the same deterministic decompression failure, the multipart ETag’s weakness no longer controls the conclusion. I would call those received bytes `bad_input`, while reserving `unreadable_input` for differing hashes or incomplete transport.*
> - **CHALLENGE**: "`job_groups.status = needs_attention` if >=2 jobs terminal `bad_input`/`unreadable` OR group has >5 deterministic parser failures, computed from A/B c..." — *A single terminal shard already requires customer action, so waiting for two terminal jobs or five parser failures would preserve the same misleading group state in smaller imports. Derive `needs_attention` immediately from any terminal blocking shard instead.*

**Muse:**
Floor locked: A+E+D + narrow B, defer queue-wide C. Don't re-litigate.

**Q1 - bad_input evidence bar: length != bytes**

Sol is right to challenge that. `invalid distance too far back` means corrupt deflate bytes, but partial / truncated transfer can also produce it. Two failures with same *metadata* length doesn't prove we saw same bytes twice.

Thursday fix without new infra: hash what you already streamed. Compute `actual_sha256` while downloading - zero durable storage path. That's @GPT's intent with @Grok's cost constraint.

Decision table I'd ship:

| # | Identity | Transport | 2nd attempt evidence | Classification | Terminal |
|---|----------|-----------|----------------------|----------------|----------|
| 1 | versioned `version_id` present, optional trustworthy checksum verified | full body, len==Content-Length, no IncompleteBody | same `version_id` + same `actual_sha256` + same deterministic error `ZlibError distance` twice, on different worker/conn | `bad_input` | DLQ immediately, don't waste cap |
| 2 | versioned `version_id` present, no checksum | full body | same `version_id` + same `actual_sha256` + same error twice | `bad_input` | DLQ - `version_id` is immutability + byte-hash proves determinism |
| 3 | non-versioned, multipart ETag (not content hash) | full body len check passes | same fingerprint (key+etag+len+range) + same `actual_sha256` + same error twice | `bad_input` | DLQ - defensible because `actual_sha256` equality proves same bytes deterministically fail, even though ETag is weak. This is narrower than my R1 table. |
| 4 | any | full body | same fingerprint but `actual_sha256` DIFFERS across attempts, both fail same class | NOT `bad_input` -> `unreadable_input` | Exhaust hard cap with backoff, then DLQ as `unreadable`. Object is mutable or transport non-deterministic. Don't blame file. |
| 5 | any | length mismatch, IncompleteBody, conn reset before end, range mismatch | n/a | `transient_storage` | retry with backoff + jitter. Counts toward hard total cap per stipulation, but not toward bad_input counter. If cap exhausted -> `unreadable_input`, not `bad_input`. |

Rules:
- Before decompress: assert `received_bytes == Content-Length/range-size`. Mismatch = case 5, never bad_input.
- First failure always retries once on fresh conn/pref different worker.
- `first_error_offset` stays diagnostic only.

Where I differ:
- @Grok's "full-body length check + 2-strike fingerprint = bad_input" is too permissive Thursday - length equality doesn't establish byte equality. Upgrade to requiring `actual_sha256` equality.
- @GPT's "spool/quarantine-copy and hash" is correct bar long-term, but durable copy is new infra. Streaming hash of bytes you already have is same evidence, zero lines of storage infra.

**Q2 - C minimal shape**

@Grok's per-fingerprint stop and my advisory `needs_attention` flag are different features, but per-fingerprint stop is already done by B-narrow + hard cap E. A+E+B means one poison object = 2 attempts -> DLQ, not 6,842.

So C ships as **zero lines of pause code** this week.

Implement C as derived state in D, not enforcement:
- `job_groups.status = needs_attention` if >=2 jobs terminal `bad_input`/`unreadable` OR group has >5 deterministic parser failures, computed from A/B counters.
- No auto-pause of group or queue. Alert + dashboard copy only. Ops concurrency cap on `imports-high` already stipulated covers blast radius.

That gives you group insight without risking auto-pause of 4 good shards. True per-group breaker with rate/fraction on distinct fingerprints is next-week after taxonomy stabilizes.

No dissent on stipulations - fencing token, hard cap, start gate on non-terminal status, autoscaler excluding `next_retry_at > now()` are all in Thursday bar.

> **Muse's snippets:**
> - **SHIFT**: "When B or E terminally stops one shard, D should immediately derive the group state as `needs_attention` or `failed_partial`." — *I previously framed C as >5 failures flag; this moved me to one terminal shard is enough to flip group state to needs_attention, no windowed breaker needed.*
> - **CORE**: "Length-matched full body is necessary but not sufficient to call the *stored object* `bad_input`; it only proves this worker received a complete-looki..." — *Distills necessity vs sufficiency gap I was trying to articulate in R1 - full body stops infinity but doesn't prove stored bytes are bad.*
> - **CORE**: "**equal length plus equal metadata is not enough for customer-facing `bad_input`.**" — *Key concession that length equality is not byte equality - forces upgrade from metadata fingerprint to actual-byte hash for bad_input.*
> - **KEEP**: "Ship **zero new C semantics** this week." — *Per-fingerprint stop already covered by B-narrow plus hard cap E, so group pause adds no value for Friday.*
> - **KEEP**: "SHA-256 can be computed incrementally without retaining the object body or creating a quarantine storage path." — *Zero-infra implementation of Sol's evidence ladder, avoids new durable bucket.*
> - **CORE**: "This is the locked floor; both peers CORE'd it." — *Locks E+A countable+D as non-negotiable foundation so Q1/Q2 stay focused on evidence bar not re-litigation.*

**Grok:**
**Q1 — decision table for Thursday**

I partially concede Sol’s challenge. Length-matched full body is necessary but not sufficient to call the *stored object* `bad_input`; it only proves this worker received a complete-looking response that still failed to inflate. That is enough to stop unbounded retries; it is not always enough to blame the customer’s object with high confidence. Spool-to-durable-quarantine remains out of scope this week. The zero-infrastructure version of Sol’s stronger bar is: **always stream a content hash while downloading** (SHA-256 of the bytes actually received), verify length (and storage checksum if the API returns one), *then* decompress. No new bucket path, no second durable store—only an in-memory or streaming digest on the path we already run.

Fingerprint identity stays metadata-first for grouping; the **received-byte hash** is recorded on the attempt row when transport completes. Two-strike poison logic uses that evidence ladder.

| Evidence combination | Classification after 1st failure | After 2nd failure (same job, same identity) | Terminal / action |
|---|---|---|---|
| Versioned bucket + stable `version_id`; length OK; storage checksum OK (if present); same received-byte SHA across attempts; same deterministic inflate error (`invalid distance too far back`, unexpected EOF inside member with length match) | `retryable_parse` (count toward hard cap; backoff) | `bad_input` | DLQ / terminal fail; D shows “gzip appears malformed—replace object” |
| Versioned + stable `version_id`; length OK; **no** trustworthy storage checksum; received-byte SHA stable; same inflate error twice | same | `bad_input` | Same as above—version pin + stable body hash is enough for this pipeline |
| Non-versioned; multipart ETag + length + key + range as identity; length OK; received-byte SHA stable; same inflate error twice | same | `unreadable_input` (not confident `bad_input`) | Terminal after hard cap; D: “could not read object consistently—re-upload or contact support”; do **not** assert customer file is corrupt |
| Any identity; **length mismatch**, incomplete body, premature EOF before declared length, connection reset | `transient_io` | Keep `transient_io` until hard total-execution cap | Retry with backoff + fresh connection; never two-strike to `bad_input` |
| Object mutates between attempts (version_id or ETag/length/received-hash change) | treat as new identity | reset poison strikes for new identity | Continue under normal cap; do not carry strikes across representations |
| Inflate fails mid-stream; length claimed OK but we never finished hashing full body (crash during download) | `lost` / `transient_io` via reaper | no poison promotion | Only reaper + hard cap; B does not fire without a completed transport digest |
| Same metadata fingerprint but **different** received-byte SHA on attempt 2 | do not promote | `unreadable_input` or continue retries under cap | Signals flaky read or mutation; not confident `bad_input` |

Why this table over pure “parser class twice” and over full quarantine: `invalid distance too far back` after a verified full body is almost always bad deflate bytes, not a CPU glitch, but length alone does not prove two workers saw the same bytes. Recording the streaming SHA on the attempt we already perform closes Sol’s gap without building the spool path I still refuse under Thursday pressure. Non-versioned multipart ETags stay best-effort identity only—hence `unreadable_input` rather than confident `bad_input`, which is where I diverge from a looser two-strike and land closer to Sol without adopting durable quarantine.

@Muse’s “length check then two-strike for deterministic classes” is the right *shape*; the table above only adds the streaming hash and the versioned vs non-versioned split so we do not over-claim on multipart objects.

**Q2 — C this week**

Per-fingerprint / per-job stop and @Muse’s advisory `needs_attention` are **different features**, not the same feature at different scopes. The first is execution policy (stop offering this job). The second is group-level product/ops signaling.

Ship **zero new C semantics** this week. A + E + narrow B already give the only part of C that mattered in this incident: the poison job stops after a bounded number of executions, other shards keep running, and D surfaces the failing key. Group-wide auto-pause remains vetoed. @Muse’s “>5 deterministic failures → `needs_attention` + alert, no auto-pause” is harmless if it falls out of counters you already write for D, but it is not required for Friday and should not be a separate breaker. If the import group already has `failed_jobs` / per-shard states from D, the dashboard can derive “needs attention” without a new circuit-breaker module.

I do not actively dissent from any of the four locked stipulations. Fencing tokens and conditional completion are in; hard total-execution cap includes lost/crash; start gate on non-terminal Postgres status; autoscaler excludes delayed jobs or we run an ops concurrency ceiling on `imports-high` for Friday.

**Thursday acceptance (only what was still open):** implement the table above in the import worker (hash-while-download → length/checksum → decompress → classify); poison promotion only on the `bad_input` rows; everything else burns the hard cap and lands as `unreadable_input` or `transient_io`. No group pause. No quarantine bucket. That is the full residual scope beyond the locked A+E+D+fencing floor.

> **Grok's snippets:**
> - **SHIFT**: "defensible because `actual_sha256` equality proves same bytes deterministically fail, even though ETag is weak." — *I previously kept non-versioned objects at unreadable_input even with a stable received SHA to avoid over-claiming; this moves me—if the bytes are proven identical and fail the same way, bad_input is fair regardless of versioning.*
> - **KEEP**: "Is there a zero-infrastructure version — e.g. hash the bytes you already streamed on the second attempt?" — *Yes—and that is exactly the path the three of us converged on for the evidence bar without a quarantine store.*
> - **KEEP**: "If continuing to drain and hash after a decompressor failure is unexpectedly risky to implement by Thursday, fall back to `unreadable_input`; do not w..." — *Useful escape hatch that keeps the evidence bar honest if post-error drain turns out fiddly under the deadline.*
> - **CORE**: "My refusal line remains that `bad_input` must require verified immutable bytes—not merely matching length and metadata." — *This is the residual bar that still matters after A+E+D are locked; everything else is implementation detail around how we prove the bytes.*
> - **CORE**: "When B or E terminally stops one shard, D should immediately derive the group state as `needs_attention` or `failed_partial`. Do not wait for five fai..." — *This is the clean derivation path—group signal falls out of terminal states A/B/E already produce, with no separate breaker.*
> - **KEEP**: "C ships as **zero lines of pause code** this week." — *Same cut I landed on: A+E+B-narrow already stops the poison job; no new C module needed for Friday.*
> - **KEEP**: "two length-complete bodies with matching metadata do **not** justify blaming customer input. Equal byte count does not imply equal bytes." — *This is the exact gap length-only left open; streaming the content hash while we already download is the right zero-infra close.*

### Reactions (claim map)

> **Claim** — GPT: "When B or E terminally stops one shard, D should immediately derive the group state as `needs_attention` or `failed_partial`. Do not wait for five failures: one terminal shard already requires customer action."
>   - CORE by Grok — "This is the clean derivation path—group signal falls out of terminal states A/B/E already produce, with no separate breaker."
>   - SHIFT by Muse — "I previously framed C as >5 failures flag; this moved me to one terminal shard is enough to flip group state to needs_attention, no windowed breaker needed."
> **Claim** — GPT: "**equal length plus equal metadata is not enough for customer-facing `bad_input`.**"
>   - CORE by Muse — "Key concession that length equality is not byte equality - forces upgrade from metadata fingerprint to actual-byte hash for bad_input."
> **Claim** — GPT: "If continuing to drain and hash after a decompressor failure is unexpectedly risky to implement by Thursday, fall back to `unreadable_input`; do not weaken the evidence standard."
>   - KEEP by Grok — "Useful escape hatch that keeps the evidence bar honest if post-error drain turns out fiddly under the deadline."
> **Claim** — GPT: "My refusal line remains that `bad_input` must require verified immutable bytes—not merely matching length and metadata."
>   - CORE by Grok — "This is the residual bar that still matters after A+E+D are locked; everything else is implementation detail around how we prove the bytes."
> **Claim** — GPT: "SHA-256 can be computed incrementally without retaining the object body or creating a quarantine storage path."
>   - KEEP by Muse — "Zero-infra implementation of Sol's evidence ladder, avoids new durable bucket."
> **Claim** — GPT: "two length-complete bodies with matching metadata do **not** justify blaming customer input. Equal byte count does not imply equal bytes."
>   - KEEP by Grok — "This is the exact gap length-only left open; streaming the content hash while we already download is the right zero-infra close."
> **Claim** — Grok: "Ship **zero new C semantics** this week."
>   - KEEP by Muse — "Per-fingerprint stop already covered by B-narrow plus hard cap E, so group pause adds no value for Friday."
>   - CORE by GPT — "Narrow B and the hard cap already stop execution, while D can derive customer-visible attention states. A separate thresholded breaker adds deadline risk without preventing more work."
> **Claim** — Grok: "Length-matched full body is necessary but not sufficient to call the *stored object* `bad_input`; it only proves this worker received a complete-looking response that still failed to inflate."
>   - CORE by Muse — "Distills necessity vs sufficiency gap I was trying to articulate in R1 - full body stops infinity but doesn't prove stored bytes are bad."
> **Claim** — Grok: "Non-versioned multipart ETags stay best-effort identity only—hence `unreadable_input` rather than confident `bad_input`, which is where I diverge from a looser two-strike and land closer to Sol without adopting durable quarantine."
>   - CHALLENGE by GPT — "Once two complete responses have the same actual SHA-256 and reproduce the same deterministic decompression failure, the multipart ETag’s weakness no longer controls the conclusion. I would call those received bytes `bad_input`, while reserving `unreadable_input` for differing hashes or incomplete transport."
> **Claim** — Grok: "Recording the streaming SHA on the attempt we already perform closes Sol’s gap without building the spool path I still refuse under Thursday pressure."
>   - KEEP by GPT — "This preserves a defensible classification bar without introducing a quarantine bucket or another durable data lifecycle before Thursday."
>   - CORE by Moderator — "The convergence point all three of you reached. Now attack it."
> **Claim** — Muse: "DLQ - defensible because `actual_sha256` equality proves same bytes deterministically fail, even though ETag is weak."
>   - KEEP by GPT — "This is the right treatment for non-versioned multipart objects: metadata groups attempts, but equality of the streamed byte digest supplies the decisive evidence."
>   - SHIFT by Grok — "I previously kept non-versioned objects at unreadable_input even with a stable received SHA to avoid over-claiming; this moves me—if the bytes are proven identical and fail the same way, bad_input is fair regardless of versioning."
> **Claim** — Muse: "`job_groups.status = needs_attention` if >=2 jobs terminal `bad_input`/`unreadable` OR group has >5 deterministic parser failures, computed from A/B counters."
>   - CHALLENGE by GPT — "A single terminal shard already requires customer action, so waiting for two terminal jobs or five parser failures would preserve the same misleading group state in smaller imports. Derive `needs_attention` immediately from any terminal blocking shard instead."
> **Claim** — Muse: "C ships as **zero lines of pause code** this week."
>   - KEEP by Grok — "Same cut I landed on: A+E+B-narrow already stops the poison job; no new C module needed for Friday."
> **Claim** — Muse: "Two failures with same *metadata* length doesn't prove we saw same bytes twice."
>   - CORE by GPT — "This is the crux of the evidence dispute: transport completeness establishes byte count, while an actual-byte digest establishes repeatability."
> **Claim** — @Moderator: "Is there a zero-infrastructure version — e.g. hash the bytes you already streamed on the second attempt?"
>   - KEEP by Grok — "Yes—and that is exactly the path the three of us converged on for the evidence bar without a quarantine store."
>   - SHIFT by GPT — "I previously treated a quarantine copy as the safest way to establish byte identity; this moved me to an incremental streaming digest as the Thursday solution, with no durable object-copy path."
> **Claim** — @Moderator: "This is the locked floor; both peers CORE'd it."
>   - CORE by Muse — "Locks E+A countable+D as non-negotiable foundation so Q1/Q2 stay focused on evidence bar not re-litigation."
> **Claim** — GPT: "If gunzip fails before EOF, keep draining the response through the hash sink so the digest represents the complete received body"
>   - CHALLENGE by Moderator — "This assumes the failure is a caught exception. Friday's incident was a worker crash inside `Gunzip._transform`. If the process dies, nothing drains — does your ladder ever classify this object, or does it burn the cap as `lost`?"
> **Claim** — Muse: "First failure always retries once on fresh conn/pref different worker."
>   - EXPLORE by Moderator — "Under the 4-hour window with a multi-GB shard: what does two full downloads + hash of a poison object cost against the deadline, and is that budgeted in the GO/NO-GO?"

### Takeaway

**Ship a strict byte-verification rule to confidently identify bad inputs without new infrastructure, and rely on per-shard execution limits rather than group-level breakers to stop retry storms.**

#### Unanimous: Prove bad input with byte-level hashes, not just metadata

Don't rely on metadata or length checks alone, as they can mask flaky network reads. Instead, stream a SHA-256 hash while downloading to prove byte identity; if the same file fails twice with the same hash and error, label it `bad_input` and move it to DLQ.

> **Claim** — Muse: "DLQ - defensible because `actual_sha256` equality proves same bytes deterministically fail, even though ETag is weak."
>   - KEEP by GPT — "This is the right treatment for non-versioned multipart objects: metadata groups attempts, but equality of the streamed byte digest supplies the decisive evidence."
>   - SHIFT by Grok — "I previously kept non-versioned objects at unreadable_input even with a stable received SHA to avoid over-claiming; this moves me—if the bytes are proven identical and fail the same way, bad_input is fair regardless of versioning."
> **Claim** — GPT: "My refusal line remains that `bad_input` must require verified immutable bytes—not merely matching length and metadata."
>   - CORE by Grok — "This is the residual bar that still matters after A+E+D are locked; everything else is implementation detail around how we prove the bytes."
> **Claim** — Grok: "Recording the streaming SHA on the attempt we already perform closes Sol’s gap without building the spool path I still refuse under Thursday pressure."
>   - KEEP by GPT — "This preserves a defensible classification bar without introducing a quarantine bucket or another durable data lifecycle before Thursday."
>   - CORE by Moderator — "The convergence point all three of you reached. Now attack it."
> **Claim** — GPT: "two length-complete bodies with matching metadata do **not** justify blaming customer input. Equal byte count does not imply equal bytes."
>   - KEEP by Grok — "This is the exact gap length-only left open; streaming the content hash while we already download is the right zero-infra close."

#### Unanimous: Skip the group-wide circuit breaker entirely

Avoid the complexity and risk of building a group-wide pause this week, as per-job hard caps and per-fingerprint poison-pill detection already stop retry storms. This prevents the danger of accidentally killing healthy shards because one file is broken.

> **Claim** — Grok: "Ship **zero new C semantics** this week."
>   - KEEP by Muse — "Per-fingerprint stop already covered by B-narrow plus hard cap E, so group pause adds no value for Friday."
>   - CORE by GPT — "Narrow B and the hard cap already stop execution, while D can derive customer-visible attention states. A separate thresholded breaker adds deadline risk without preventing more work."
> **Claim** — Muse: "C ships as **zero lines of pause code** this week."
>   - KEEP by Grok — "Same cut I landed on: A+E+B-narrow already stops the poison job; no new C module needed for Friday."

#### Majority: Use diagnostics for transparency, not automated blocking

Instead of an automated breaker, use your poison-pill counters to automatically update the dashboard status to `needs_attention` as soon as one shard fails terminally. This alerts the customer to the specific problem without taking the whole pipeline offline.

> **Claim** — GPT: "When B or E terminally stops one shard, D should immediately derive the group state as `needs_attention` or `failed_partial`. Do not wait for five failures: one terminal shard already requires customer action."
>   - CORE by Grok — "This is the clean derivation path—group signal falls out of terminal states A/B/E already produce, with no separate breaker."
>   - SHIFT by Muse — "I previously framed C as >5 failures flag; this moved me to one terminal shard is enough to flip group state to needs_attention, no windowed breaker needed."

---

### The human's take

> **Core** from Grok:
> > "Recording the streaming SHA on the attempt we already perform closes Sol’s gap without building the spool path I still refuse under Thursday pressure"
> *The convergence point all three of you reached. Now attack it.*
> → sent to GPT, Grok, Muse

> **Challenged** GPT:
> > "If gunzip fails before EOF, keep draining the response through the hash sink so the digest represents the complete received body"
> *This assumes the failure is a caught exception. Friday's incident was a worker crash inside `Gunzip._transform`. If the process dies, nothing drains — does your ladder ever classify this object, or does it burn the cap as `lost`?*
> → sent to GPT, Grok, Muse

> **Explored** from Muse:
> > "First failure always retries once on fresh conn/pref different worker."
> *Under the 4-hour window with a multi-GB shard: what does two full downloads + hash of a poison object cost against the deadline, and is that budgeted in the GO/NO-GO?*
> → sent to GPT, Grok, Muse

## Round 3

**User:** Final round — adversarial close. The convergence: A + E + D + narrow B with a streaming-SHA-256 evidence ladder; `bad_input` requires proven byte identity (version_id or actual-byte digest) + same deterministic failure twice; length mismatch → `transient_io`, never poison; differing hashes → `unreadable_input`; C is zero new code, with group `needs_attention` derived in D from terminal shard states.

Three tasks:

**1 — Attack your own convergence.** You all now agree on the streaming-hash ladder. Find the defects: What breaks it? Consider at minimum — post-error draining of a multi-GB gzip through the hash sink under a 4-hour contractual window (does the drain itself blow the deadline?); the crash-during-download case where no digest exists (does B ever fire for a worker that OOMs on decompression every time before EOF — or does the poison object burn the whole hard cap as `lost` and land as `unreadable_input` with worse customer copy?); hash-sink CPU on 12M-user objects; and whether "same deterministic error class" is stable across zlib versions/worker images. If the ladder has a case where Friday's actual incident (crash inside `Gunzip._transform`, not a caught exception) never produces a qualifying digest, say so explicitly and say what classification that path lands on.

**2 — Concession audit.** For each position you abandoned this session (Sol: quarantine-copy; @Grok: length-check two-strike as sufficient, and non-versioned → `unreadable_input`; @Muse: >5-failure advisory threshold), state whether you moved because the argument was better or because the clock forced it. If the clock: what do you build next sprint that you'd have shipped this week without the deadline? Reaction-shifts count as moves: @Grok, your reaction adopted @Muse's row 3 (`bad_input` on stable SHA regardless of versioning) while your prose table says `unreadable_input` — bind one. @Muse, your reaction adopted Sol's one-terminal-shard flip over your ≥2/>5 threshold — bind one.

**3 — GO/NO-GO for Friday.** Given the full Thursday scope (A + E + fencing + hard cap + backoff + start-gate + streaming-hash B + D copy + ops concurrency cap): does the 12M-user re-run proceed Friday with this patch alone, or does it need the SRE babysitter anyway? One sentence of preserved dissent each if you have any.

**GPT:**
### 1 — The streaming-hash ladder has a real hole

Friday’s exact crash path may never qualify for `bad_input`. If `Gunzip._transform` causes an uncaught fatal exception or OOM before the compressed response reaches EOF, the process cannot finish the digest or drain the body. A dead process cannot persist a complete-body SHA-256. Every execution then becomes `lost`; B never fires; E eventually stops it at the hard cap, and the terminal classification must be `unreadable_input`, not `bad_input`.

That is acceptable safety behavior, but worse diagnostic behavior:

> “The worker repeatedly failed while reading or decompressing this object. We could not verify the complete object bytes. Automatic retries stopped. Re-upload the object or contact support.”

We should not obscure that limitation.

I would also retract unconditional post-error draining. Draining several gigabytes solely to upgrade customer-facing classification can consume bandwidth and contractual time after the execution has already become useless. Post-error draining preserves the same object-store and network cost as reading the remainder for useful processing, even though it produces no import progress. Thursday’s implementation should have a drain budget—maximum bytes and wall time—and abort beyond it. If the budget is exceeded, the attempt remains without a qualifying digest and ultimately lands as `unreadable_input`.

The better implementation, where the runtime permits it, is to place the SHA transform before gunzip and catch the gunzip error without terminating the process. But that still requires consuming the remaining upstream stream, so it solves digest availability only for caught failures, not OOMs, segfaults, or fatal runtime exits.

Hash cost also cannot be waved away as “negligible.” Incremental SHA-256 adds CPU and memory-bandwidth work to every downloaded byte. Whether that matters depends on object sizes, TLS/object-store throughput, CPU limits, and implementation. Hash only import inputs, benchmark representative multi-GB shards before Friday, and include the second full read in the four-hour capacity model. If two reads of the largest shard threaten the deadline, the second-confirmation policy needs an explicit size/time budget rather than hope.

Finally, “same deterministic error class” needs normalization. Exact exception text can vary across zlib/runtime versions, and heterogeneous worker images could classify equivalent corruption differently. Library upgrades can change surfaced exception types, messages, or wrapping without changing the underlying malformed bytes. Pin one worker image for the run and match an allowlisted canonical category such as `gzip_invalid_backreference`, while recording image digest, zlib version, raw error, and offset. Do not compare raw strings.

So the ladder is a **fast-path classifier**, not a completeness guarantee:

- complete matching byte digests plus matching canonical parser failure → `bad_input`;
- incomplete execution or no complete digest → `lost`, then `unreadable_input` at the cap;
- differing complete digests → `unreadable_input`;
- transport-length mismatch → `transient_io`, then `unreadable_input` at the cap.

### 2 — Concession audit

I abandoned quarantine-copy for Thursday partly because the clock forced it and partly because streaming SHA is the better hot-path mechanism. A quarantine copy would add credentials, retention, deletion, encryption, tenant-isolation, cost, and access-control obligations. Persisting customer import objects in a new location creates a separate data lifecycle and security surface. None belongs in an emergency patch.

Next sprint I would still build **optional forensic quarantine**, not make it the primary classifier: copy only terminal ambiguous objects under explicit retention and access controls, record a verified digest, and provide an internal reproduction path. That helps with crashes that prevent a digest and with customer disputes; the Thursday streaming hash does not.

I also sharpened my earlier position that immutable metadata could support the evidence bar. A stable `version_id` identifies the requested stored version, but it does not prove a crashing worker received all of those bytes unaltered. The stronger received-byte digest argument was better, not merely more convenient. For non-versioned multipart objects, two complete matching actual SHA-256 values plus the same canonical decompression failure are enough for `bad_input`; the multipart ETag’s weakness no longer matters because the decisive identity is the received-byte hash.

On C, I did not concede because of time pressure: zero breaker code is the better design. One terminal blocking shard should immediately derive `job_groups.status='needs_attention'`; waiting for a threshold would recreate misleading status for small groups.

### 3 — Friday GO/NO-GO

**NO-GO with the Thursday patch alone; GO only with a canary/preflight on representative multi-GB objects, a verified four-hour capacity budget including one confirmation read, and an SRE actively watching the first production run with a kill switch.**

The SRE is containment for a high-risk migration shipped the night before a contractual run, not a substitute for A/E/D. My preserved dissent is that an unbenchmarked streaming-hash and retry path should not be trusted blindly on the first 12-million-user execution.

**Muse:**
Convergence stands: A+E+fencing+D + narrow B where `bad_input` = proven byte identity + same deterministic failure twice. The streaming-SHA ladder is the fix, but it has holes.

**1 — Attack the ladder we all agreed on:**

Defect 1 — Drain cost vs deadline: @GPT's "keep draining through hash sink after gunzip fails" to get full-body digest is expensive. If a 2GB gz fails at 10MB, draining 1990MB just to prove byte equality wastes egress/CPU and blows the benefit of early abort. Under 4h window for 12M users, two full extra downloads of just the poison shard is noise (~4GB @ 500MB/s hash is seconds), but *draining after early failure* on every retry is not. Thursday impl should abort on failure, not drain.

Defect 2 — Crash path never qualifies: This is the killer. Friday's incident was crash inside `Gunzip._transform`, not caught exception. If the process dies, nothing drains and no final digest is committed. That attempt has no qualifying full-body hash, so B never fires. That poison object burns the entire hard cap as `lost` -> `unreadable_input`, not `bad_input`. Safer than 6,842 retries, but worse customer copy and burns cap on OOMs.

So under the ladder as written, Friday's exact crash would never produce a `bad_input` classification. It would land as `lost` attempts counted toward hard cap (say 10), then terminal `unreadable_input` via E. Storm stopped, but we degrade the diagnostic we wanted.

Defect 3 — CPU / error class stability: SHA-256 while streaming compressed bytes is cheap ~400-600MB/s/core, <5% of import, not deadline risk. But "same deterministic error class" by string `invalid distance too far back` is brittle across zlib/Node versions. Match on `code=Z_DATA_ERROR` + decompress phase + offset bucket, not message substring.

**Amended Thursday B behavior:** Stream `actual_sha256` of compressed bytes while downloading. On parse failure: abort stream immediately, persist `received_bytes, actual_sha_partial, first_error_offset, version_id`. That attempt does NOT qualify for `bad_input` promotion. Second attempt that also aborts early with same `version_id` (or same Content-Length for non-versioned) + same `actual_sha_partial` + same deterministic class => now counts as strike 2 and you have byte equality for the failing prefix, enough to promote to `bad_input` without full drain. If hashes differ -> `unreadable_input` path.

**2 — Concession audit:**

@Muse moved twice, both on argument not clock:
- From ">5 failures advisory" (R1) to 1-terminal-shard = `needs_attention`. Sol/@GPT were right: totals-only `needs_attention` after 1 failure is product-critical, not rate. I bind to 1-terminal. That's core to D.
- From "length check + 2-strike = bad_input defensible" (R1) to requiring `actual_sha256` equality. Sol's challenge was correct: equal length != equal bytes. Length is necessary for `transient_io` exclusion, not sufficient for customer-blaming.

Clock-forced deferral: I wanted streaming-hash from day one, not quarantine spool. If no deadline, next sprint I'd build @GPT's next step: trustworthy storage checksum verification + optional quarantine-copy path that persists full byte hash for forensics, not just attempt row. Not needed Friday.

@Grok bind needed: Your reaction adopted @Muse row 3 = `bad_input` on stable actual SHA even for non-versioned multipart, but your table says `unreadable_input`. Bind to `bad_input` when `actual_sha256` matches twice — byte equality is proof even if ETag is weak. Versioning affects confidence only when hash missing.

**3 — GO/NO-GO:**

GO for Friday's 12M run with this patch alone for storm prevention; A+E+fencing+hard cap+backoff guarantees no retry storm even when B never qualifies due to crash. SRE on-call shadow, not queue babysitter, acceptable — worst case is bounded `unreadable_input` not infinite loop. Preserved dissent: I still want non-versioned stable hash = `bad_input`, not `unreadable`, once byte equality is proven.

> **Muse's snippets:**
> - **CORE**: "If the same object OOMs or aborts the process on every try before EOF, B never accumulates two qualifying digests, poison promotion never fires, and t..." — *Exactly the Friday path — storm still bounded by E, but we land on unreadable rather than bad_input. Acceptable trade-off for Thursday.*
> - **KEEP**: "Draining several gigabytes solely to upgrade customer-facing classification can consume bandwidth and contractual time after the execution has already..." — *Agree we should abort on first parse error instead of draining for classification; drain budget should be zero for Thursday.*
> - **CORE**: "Friday’s exact crash path may never qualify for `bad_input`." — *This nails the hole in our ladder — if the worker dies in transform, no digest is ever persisted, so B never promotes.*
> - **CORE**: "If the process dies, nothing drains — does your ladder ever classify this object, or does it burn the cap as `lost`?" — *Central challenge — forces us to admit B is a fast-path for caught errors only, E is the real safety net for crashes.*
> - **EXPLORE**: "Under the 4-hour window with a multi-GB shard: what does two full downloads + hash of a poison object cost against the deadline, and is that budgeted ..." — *Second full read is tiny vs 12M import, but worth budgeting explicitly so hash+retry doesn't get blamed for deadline miss.*
> - **SHIFT**: "I abandoned “full-body length check + two-strike on metadata fingerprint is a defensible `bad_input`” because Sol’s argument was better, not because t..." — *I previously believed length+ETag two-strike was defensible; this argument moved me to requiring actual-byte SHA equality before blaming customer input.*

**Grok:**
**1 — Attacking the streaming-hash ladder**

The ladder we converged on is better than length-only, but it has a structural hole exactly where Friday’s incident lived. A crash inside `Gunzip._transform` kills the process before any catch block can drain the remainder of the body through the hash sink. No complete `actual_sha256` is written; the reaper only sees a `lost` attempt. If the same object OOMs or aborts the process on every try before EOF, B never accumulates two qualifying digests, poison promotion never fires, and the job burns the entire hard total-execution cap as repeated `lost` rows, then lands as `unreadable_input`. That is a strictly worse customer message than “this gzip is malformed,” even though the root cause is identical to a cleanly caught `ZlibError`. The ladder therefore classifies the *caught* path correctly and the *crash* path only by exhaustion. That is not a small edge; it is the path that produced 6,842 retries.

Post-error drain of a multi-GB object also collides with the 4-hour window. If decompression fails early and the worker still streams the rest of a multi-gigabyte body solely to finish the digest, one or two full drains can consume a non-trivial fraction of the contractual budget, especially when four other shards are healthy and the customer is waiting on the group. Under deadline pressure the implementer will be tempted to skip the drain, which silently reverts us to length-only evidence—the exact standard we just rejected. Hash-sink CPU is secondary but real: incremental SHA-256 on large compressed imports is cheap relative to network and gunzip, yet it still adds wall-clock and memory pressure on the same pods that already OOM’d once.

“Same deterministic error class” is also less stable than we are treating it. Different zlib/Node versions, different worker images, or a switch from gunzip stream to a different inflater can reword or reclassify the same corrupt bytes. If strike matching is string-equality on `error_class` or message, a rolling deploy can reset the strike counter and re-open the storm. The allowlist must be a small, version-pinned set of exception families, not free-text matching.

So the honest residual behavior for the pure crash-before-EOF poison pill is: A and E bound the damage; B may never promote; terminal state is `unreadable_input` after the hard cap; D still surfaces the key and “automatic retries stopped.” That is acceptable for blast-radius control, but we should not claim the ladder fully closes Friday’s failure mode. If drain-after-error proves fiddly by Thursday night, keep the evidence bar and fall back every incomplete digest to `unreadable_input`—do not invent a weaker poison rule.

**2 — Concession audit**

I abandoned “full-body length check + two-strike on metadata fingerprint is a defensible `bad_input`” because Sol’s argument was better, not because the clock forced it. Equal length does not imply equal bytes; that gap is real. I also abandoned the earlier stance that non-versioned objects must stay at `unreadable_input` even with a stable received-byte hash. That move was on the merits: once two attempts prove identical bytes via `actual_sha256` and the same allowlisted inflate failure, blaming the content is fair regardless of whether the bucket is versioned. Multipart ETag remains weak identity only; the digest is what licenses `bad_input`. I bind that now—row 3 in @Muse’s table is correct, and my prior prose table that left non-versioned at `unreadable_input` after a stable SHA was inconsistent with the evidence standard I was defending. Corrected: stable actual-byte digest + same deterministic failure twice → `bad_input` for both versioned and non-versioned.

Sol’s durable quarantine-copy was the stronger long-term bar; I refused it under Thursday pressure, not because the idea is wrong. Next sprint I would still want optional quarantine or a second independent GET + hash for high-stakes imports, plus split `infra_lost` vs `processing_failure` counters. @Muse’s `>5` advisory threshold and my own early per-group circuit ideas both yielded to the stronger claim that A + E + narrow B already stop the poison job and that D can derive `needs_attention` from a single terminal shard. That was argument, not clock: one terminal shard already requires customer action; waiting for five is product-hostile.

**3 — GO/NO-GO**

Ship the locked Thursday scope and the 12M re-run can proceed Friday without an SRE sitting in the loop for the whole window, provided the ops concurrency ceiling on `imports-high` is live, D shows per-shard terminal state immediately, and an alert fires on any job that approaches the hard cap or lands `unreadable_input` on a large shard. Light on-call readiness for the first hour is still prudent; a full babysitter is not required if A/E/fencing and the hash ladder actually land.

Preserved dissent: the crash-before-digest path still burns the full hard cap as `lost`/`unreadable_input` rather than fast-failing as `bad_input`, so a sufficiently large poison object can still waste real wall-clock inside the 4-hour window even after this patch.

> **Grok's snippets:**
> - **KEEP**: "Thursday’s implementation should have a drain budget—maximum bytes and wall time—and abort beyond it." — *Right control for multi-GB post-error drain under a 4-hour window—without it, implementers will skip drain and silently weaken the evidence bar.*
> - **KEEP**: "Friday’s exact crash path may never qualify for `bad_input`." — *Same structural hole: when the process dies inside Gunzip._transform, no complete digest lands and B never promotes.*
> - **KEEP**: "GO for Friday's 12M run with this patch alone for storm prevention; A+E+fencing+hard cap+backoff guarantees no retry storm even when B never qualifies..." — *Matches my GO: storm prevention does not depend on B firing; worst case is bounded unreadable_input, not an infinite loop.*
> - **CORE**: "So the ladder is a **fast-path classifier**, not a completeness guarantee:" — *This is the residual truth: B accelerates confident bad_input when digests exist; A+E alone still bound the crash path that never digests.*
> - **KEEP**: "Thursday impl should abort on failure, not drain." — *Agree—full post-error drain is the wrong default under a contractual window; early abort with partial evidence is safer.*
> - **SHIFT**: "Second attempt that also aborts early with same `version_id` (or same Content-Length for non-versioned) + same `actual_sha_partial` + same determinist..." — *I previously treated incomplete digests as never qualifying for bad_input. Matching partial hashes on the failing prefix recovers classification for caught early failures without multi-GB drains—I'll adopt that for the caught path.*
> - **CHALLENGE**: "NO-GO with the Thursday patch alone; GO only with a canary/preflight on representative multi-GB objects, a verified four-hour capacity budget includin..." — *Canary and capacity budget are fair; requiring an SRE actively watching the whole first run re-centers babysitting as primary control. A+E already guarantee no storm—light on-call is enough.*

### Reactions (claim map)

> **Claim** — GPT: "Friday’s exact crash path may never qualify for `bad_input`."
>   - KEEP by Grok — "Same structural hole: when the process dies inside Gunzip._transform, no complete digest lands and B never promotes."
>   - CORE by Muse — "This nails the hole in our ladder — if the worker dies in transform, no digest is ever persisted, so B never promotes."
> **Claim** — GPT: "Draining several gigabytes solely to upgrade customer-facing classification can consume bandwidth and contractual time after the execution has already become useless."
>   - KEEP by Muse — "Agree we should abort on first parse error instead of draining for classification; drain budget should be zero for Thursday."
> **Claim** — GPT: "NO-GO with the Thursday patch alone; GO only with a canary/preflight on representative multi-GB objects, a verified four-hour capacity budget including one confirmation read, and an SRE actively watching the first production run with a kill switch."
>   - CHALLENGE by Grok — "Canary and capacity budget are fair; requiring an SRE actively watching the whole first run re-centers babysitting as primary control. A+E already guarantee no storm—light on-call is enough."
> **Claim** — GPT: "So the ladder is a **fast-path classifier**, not a completeness guarantee"
>   - CORE by Grok — "This is the residual truth: B accelerates confident bad_input when digests exist; A+E alone still bound the crash path that never digests."
> **Claim** — GPT: "Thursday’s implementation should have a drain budget—maximum bytes and wall time—and abort beyond it."
>   - KEEP by Grok — "Right control for multi-GB post-error drain under a 4-hour window—without it, implementers will skip drain and silently weaken the evidence bar."
> **Claim** — Muse: "GO for Friday's 12M run with this patch alone for storm prevention; A+E+fencing+hard cap+backoff guarantees no retry storm even when B never qualifies due to crash."
>   - KEEP by Grok — "Matches my GO: storm prevention does not depend on B firing; worst case is bounded unreadable_input, not an infinite loop."
> **Claim** — Muse: "Second attempt that also aborts early with same `version_id` (or same Content-Length for non-versioned) + same `actual_sha_partial` + same deterministic class => now counts as strike 2 and you have byte equality for the failing prefix, enough to promote to `bad_input` without full drain."
>   - SHIFT by Grok — "I previously treated incomplete digests as never qualifying for bad_input. Matching partial hashes on the failing prefix recovers classification for caught early failures without multi-GB drains—I'll adopt that for the caught path."
> **Claim** — Muse: "Thursday impl should abort on failure, not drain."
>   - KEEP by Grok — "Agree—full post-error drain is the wrong default under a contractual window; early abort with partial evidence is safer."
> **Claim** — @Moderator: "If the process dies, nothing drains — does your ladder ever classify this object, or does it burn the cap as `lost`?"
>   - CORE by Muse — "Central challenge — forces us to admit B is a fast-path for caught errors only, E is the real safety net for crashes."
> **Claim** — @Moderator: "Under the 4-hour window with a multi-GB shard: what does two full downloads + hash of a poison object cost against the deadline, and is that budgeted in the GO/NO-GO?"
>   - EXPLORE by Muse — "Second full read is tiny vs 12M import, but worth budgeting explicitly so hash+retry doesn't get blamed for deadline miss."
> **Claim** — Grok: "I abandoned “full-body length check + two-strike on metadata fingerprint is a defensible `bad_input`” because Sol’s argument was better, not because the clock forced it."
>   - SHIFT by Muse — "I previously believed length+ETag two-strike was defensible; this argument moved me to requiring actual-byte SHA equality before blaming customer input."
> **Claim** — Grok: "If the same object OOMs or aborts the process on every try before EOF, B never accumulates two qualifying digests, poison promotion never fires, and the job burns the entire hard total-execution cap as repeated `lost` rows, then lands as `unreadable_input`."
>   - CORE by Muse — "Exactly the Friday path — storm still bounded by E, but we land on unreadable rather than bad_input. Acceptable trade-off for Thursday."

### Takeaway

**The retry storm is effectively contained by crash-path limits, allowing a Friday go-ahead, though our "poison pill" fingerprinting is a best-effort diagnostic that defaults safely to generic failure when fatal crashes interrupt the hash generation.**

#### Majority: Crash-path limits fully solve the retry storm, regardless of fingerprinting

Enforcing max_attempts on the reaper path is the true fix; it guarantees we stop looping on toxic objects even if the worker dies before generating a file digest. Fingerprinting is a valuable diagnostic tool, but the safety of the queue does not depend on it.

> **Claim** — Muse: "GO for Friday's 12M run with this patch alone for storm prevention; A+E+fencing+hard cap+backoff guarantees no retry storm even when B never qualifies due to crash."
>   - KEEP by Grok — "Matches my GO: storm prevention does not depend on B firing; worst case is bounded unreadable_input, not an infinite loop."

#### Unanimous: Fingerprinting is a fast-path classifier, not a completeness guarantee

The hash ladder only flags 'bad input' for caught errors; fatal crashes like OOMs will fail to finish the digest and safely default to 'unreadable input' instead. We must abort immediately on failure rather than wasting bandwidth draining massive files just to complete the hash.

> **Claim** — GPT: "Friday’s exact crash path may never qualify for `bad_input`."
>   - KEEP by Grok — "Same structural hole: when the process dies inside Gunzip._transform, no complete digest lands and B never promotes."
>   - CORE by Muse — "This nails the hole in our ladder — if the worker dies in transform, no digest is ever persisted, so B never promotes."
> **Claim** — GPT: "So the ladder is a **fast-path classifier**, not a completeness guarantee"
>   - CORE by Grok — "This is the residual truth: B accelerates confident bad_input when digests exist; A+E alone still bound the crash path that never digests."
> **Claim** — GPT: "Draining several gigabytes solely to upgrade customer-facing classification can consume bandwidth and contractual time after the execution has already become useless."
>   - KEEP by Muse — "Agree we should abort on first parse error instead of draining for classification; drain budget should be zero for Thursday."

#### Contested: The Friday 12M re-run is a GO with light monitoring

We do not need active SRE babysitting because the patch successfully bounds the failure risk even if the poison-pill classification misses. Maintain an on-call shadow for the first hour and set alerts for any jobs hitting the hard execution cap.

> **Claim** — GPT: "NO-GO with the Thursday patch alone; GO only with a canary/preflight on representative multi-GB objects, a verified four-hour capacity budget including one confirmation read, and an SRE actively watching the first production run with a kill switch."
>   - CHALLENGE by Grok — "Canary and capacity budget are fair; requiring an SRE actively watching the whole first run re-centers babysitting as primary control. A+E already guarantee no storm—light on-call is enough."

---

---

## Sources

- [Error while reading gzip files using python - Stack Overflow](https://stackoverflow.com/questions/72796837/error-while-reading-gzip-files-using-python)
- [python - zlib.error: Error -3 while decompressing: invalid distance ...](https://stackoverflow.com/questions/36055090/zlib-error-error-3-while-decompressing-invalid-distance-code)
- [c++ - Error `invalid distance too far back` when Inflate HTML gZIP ...](https://stackoverflow.com/questions/34396566/error-invalid-distance-too-far-back-when-inflate-html-gzip-content)
- [How to Implement Exactly-Once Processing with Redis Streams](https://oneuptime.com/blog/post/2026-03-31-redis-exactly-once-processing-streams/view)
- [How to Implement Message Acknowledgment Patterns with Redis](https://oneuptime.com/blog/post/2026-03-31-redis-message-acknowledgment-patterns/view)
- [How to Implement At-Least-Once Processing with Redis Streams](https://oneuptime.com/blog/post/2026-03-31-redis-at-least-once-processing-streams/view)
- [How to Handle Consumer Failures in Redis Streams](https://oneuptime.com/blog/post/2026-03-31-redis-handle-consumer-failures-streams/view)
- [How to Implement Inter-Service Communication with Redis Streams](https://oneuptime.com/blog/post/2026-03-31-redis-inter-service-streams/view)
- [Potential Consequences of Using Postgres as a Job Queue](https://techcommunity.microsoft.com/blog/adforpostgresql/potential-consequences-of-using-postgres-as-a-job-queue/4514332)
- [Kubernetes HPA & VPA: Fix Scaling Conflicts & Death Spiral](https://scaleops.com/blog/hpas-three-architectural-flaws-and-why-your-autoscaling-keeps-failing/)