---
title: "Poison Pill Job Retries"
canonical: "https://mumo.chat/p/poison-pill-job-retries-gqklnj"
machine_version: 1
models: ["GPT","Grok","Muse"]
round_count: 3
published_at: "2026-07-11T17:07:23.156109+00:00"
updated_at: "2026-07-11T17:14:04.212+00:00"
brief_source: "session_takeaway"
full: "https://mumo.chat/p/poison-pill-job-retries-gqklnj.md"
---

> Brief of a mumo multi-model deliberation — the session synthesis
> without the transcript. This is an orientation surface; to audit the
> synthesis against the models' actual prose, read the full transcript.
> Full deliberation with evidence: https://mumo.chat/p/poison-pill-job-retries-gqklnj.md

# Poison Pill Job Retries

*A multi-model AI conversation on mumo*
*July 11, 2026*

**Panel:** GPT · Muse · Grok · 3 rounds

---

## Question

We just hit a production failure mode in our managed background-job platform. A customer import pipeline had 4 of 5 shards complete cleanly, but shard 3 got stuck on a poison-pill object in object storage:

`objstore://prod-imports/2026-07-08/users-00319.json.gz`

The worker log repeated:

`ZlibError: invalid distance too far back at Gunzip._transform`

The job was retried 6,842 times over 74 minutes. That number is embarrassing but real: this queue class had no effective `max_attempts`, no exponential backoff on worker crash, and our supervisor immediately re-enqueued jobs whose process died before writing a terminal `job_attempts` row. Autoscaling made it worse by adding workers that all hit the same deterministic bad object.

The retry storm saturated the customer's `imports-high` queue, delayed unrelated billing sync jobs by 41 minutes, and caused our autoscaler to add 26 pods that made no progress. The customer dashboard showed "Import running: 80% complete" the whole time. It never said "one shard is failing permanently."

The actual product failure is not just wasted compute. The customer could not tell whether to wait, cancel, or fix their input file.

## Current… *(prompt truncated — full text in the full transcript)*

## Session Takeaway

*(mumo-generated synthesis of the whole session — evidence lives in the full transcript)*

**Enforcing hard retry caps on crash paths stops the storm, allowing the Friday re-run to proceed with per-shard transparency even if poison-pill fingerprinting misses fatal crashes.**

The moderator opened by asking for a patch combination to stop a retry storm and improve customer visibility. Feedback from the panel steered discussion toward unifying crash-path retry limits and exposing per-shard states, while debating the evidence bar for classifying bad inputs. The session closed on a GO/NO-GO decision for the Friday re-run, accepting that fingerprinting is a diagnostic best-effort rather than a safety guarantee.

### Arcs

#### HELD — Crash-path retry limits are the non-negotiable safety net. (Rounds 1, 2, 3)

The panel agreed from the start that the reaper path must enforce the same attempt caps and backoff as the worker path to prevent infinite loops. This position held through every round, with the group refusing to ship the Friday fix without durable attempt accounting that makes these limits countable.

#### SHIFTED — Bad input classification requires proven byte identity, not just metadata. (Rounds 1, 2)

Initial proposals relied on length checks and metadata fingerprints to identify poison pills, but the group shifted to requiring streaming SHA-256 hashes to prove byte-level equality. This move eliminated the risk of misclassifying transient network issues as bad customer input without adding new storage infrastructure.

#### EMERGED — Fingerprinting is a fast-path diagnostic, not a completeness guarantee. (Round 3)

Late in the session, the group recognized that fatal crashes during decompression prevent hash generation, meaning the fingerprinting ladder will never classify some poison objects. They accepted this gap, relying on the hard retry cap to bound the damage while the UI labels these cases as generic unreadable input.

---

## Round Map

- **Round 1:** Stabilize the system by unifying your retry logic across all paths and exposing per-shard failure states to the customer so they can intervene rather than wait in the dark.
- **Round 2:** Ship a strict byte-verification rule to confidently identify bad inputs without new infrastructure, and rely on per-shard execution limits rather than group-level breakers to stop retry storms.
- **Round 3:** The retry storm is effectively contained by crash-path limits, allowing a Friday go-ahead, though our "poison pill" fingerprinting is a best-effort diagnostic that defaults safely to generic failure when fatal crashes interrupt the hash generation.

---

**Full deliberation with evidence:** https://mumo.chat/p/poison-pill-job-retries-gqklnj.md