---
title: "LLM Note Worker Hang"
canonical: "https://mumo.chat/p/llm-note-worker-hang-dba5ta"
machine_version: 1
models: ["GPT","Grok","Muse"]
round_count: 3
published_at: "2026-07-11T17:07:23.156109+00:00"
updated_at: "2026-07-11T17:13:50.257+00:00"
brief_source: "session_takeaway"
full: "https://mumo.chat/p/llm-note-worker-hang-dba5ta.md"
---

> Brief of a mumo multi-model deliberation — the session synthesis
> without the transcript. This is an orientation surface; to audit the
> synthesis against the models' actual prose, read the full transcript.
> Full deliberation with evidence: https://mumo.chat/p/llm-note-worker-hang-dba5ta.md

# LLM Note Worker Hang

*A multi-model AI conversation on mumo*
*July 11, 2026*

**Panel:** GPT · Muse · Grok · 3 rounds

---

## Question

We just hit a production failure mode in our ambient clinical-note product, used by 14 outpatient clinics. A batch of 38 encounter recordings from one large family-medicine clinic finished speech-to-text cleanly, but the note-generation worker (via our managed LLM endpoint) hung mid-stream on encounter `enc_7f2c…` until our container watchdog (a container-platform task stop via health-check + 15-minute supervisor timeout — there is no serverless-style platform maxDuration here) killed the task. That encounter’s `clinical_notes` row was never pre-inserted. From the provider’s view in the EHR plugin, the encounter shows “Transcribed · Note pending…” forever.

Two other encounters in the same queue batch completed inside that task before the hang. The remaining messages in the in-flight batch became visible again only after the visibility timeout; meanwhile the clinic dashboard still read aggregate “Import/note batch running: ~80%” because `job_groups` only stores totals, not per-encounter terminal failure. Unrelated billing sync on the same queue namespace waited on concurrency limits while the autoscaler added workers that re-hit the same hung prompt shape.

The actual product… *(prompt truncated — full text in the full transcript)*

## Session Takeaway

*(mumo-generated synthesis of the whole session — evidence lives in the full transcript)*

**Ship durable pre-attempt tracking, per-encounter task isolation, and passive circuit breaking by Thursday, but discard shape-based poison detection to avoid false positives during peak clinic hours.**

The moderator opened by asking for a triage of four architectural options to fix a silent LLM hang in clinical note generation. Feedback from the panel steered discussion toward rejecting active pre-flight probes in favor of durable state tracking and passive circuit breaking, then forced a decision on batch isolation under tight deadlines. The session closed by settling the final poison-detection rule and validating the data-integrity fencing needed to ship the patch by Thursday.

### Arcs

#### HELD — Durable pre-attempt state is the mandatory foundation for visibility. (Rounds 1, 2)

The panel agreed from the start that you must record an attempt in the database before calling the LLM, ensuring that a watchdog kill leaves a persistent 'lost' record rather than a silent hole. This transactional link remains the non-negotiable prerequisite for making failures observable and terminal.

#### SHIFTED — Full task isolation ships Thursday as an operational necessity, not a security fix. (Rounds 1, 2)

Initial hesitation about the orchestration cost vanished once the panel confirmed it was a simple templating change, but the justification shifted from HIPAA blast-radius reduction to pure failure-domain isolation. The +18% compute premium is now explicitly defended as the price of preventing one poisoned prompt from stranding an entire clinic batch.

#### EMERGED — Poison detection must use exact-request fingerprinting, not coarse shape buckets. (Rounds 2, 3)

A mid-session proposal to quarantine encounters based on similar failure shapes was rejected for creating too many false positives at low clinic volumes. The final rule fingerprints exact requests and only triggers when the broader fleet is healthy, ensuring that provider-wide outages don't permanently defer good encounters.

---

## Round Map

- **Round 1:** Implement durable pre-attempt state and per-task isolation to make failures observable and terminal, and reject per-request probes that threaten capacity during peak hours.
- **Round 2:** The team is shipping full task isolation (C), durable fenced generation (D), and a passive circuit breaker by Thursday, explicitly justifying the extra compute as a tool for operational clarity rather than security.
- **Round 3:** GO for Thursday with the D+C+A architecture, but manage the "loudness" of your new failure reporting carefully to prevent a secondary capacity crunch during peak clinic hours.

---

**Full deliberation with evidence:** https://mumo.chat/p/llm-note-worker-hang-dba5ta.md