LLM Note Worker Hang

I would ship D + C + a focused version of A, and explicitly reject B in the request path.

D is mandatory groundwork. Pre-create the encounter-level note lifecycle row and the attempt row before any model call. Do not hold a database transaction open during inference: atomically claim the encounter, upsert clinical_notes(note_status='generating'), and insert note_attempts(status='running', lease_expires_at=…, attempt_no=…), then commit and call the provider. Without durable state established before inference, every timeout, SIGKILL, or node loss can recreate today’s invisible-null-row failure.

The state machine should be explicit rather than inferred from nullable timestamps:

queued → generating → ready
generating → retry_scheduled → generating
generating → lost → retry_scheduled | failed | deferred_to_human

A reaper should mark expired leases lost; termination handlers are only best-effort. A process killed by the platform cannot be relied upon to record its own failure. Put a uniqueness constraint on the logical note/version and use a stable attempt or generation key where the provider supports idempotency. There is always an ambiguity window where inference may have completed but the worker died before committing success, so “exactly once” cannot come from the worker alone.

I would also ship C as one encounter per task, not merely small shards, for this clinic-facing path. The primary benefit is failure-domain isolation, not HIPAA. One poisoned prompt will no longer strand unrelated encounters or leave an entire received batch invisible behind one worker. It also reduces the amount of PHI simultaneously resident in one process, but it does not materially reduce the authorization or infrastructure blast radius if every task retains the same IAM permissions, network access, and logging configuration. The estimated 18% compute premium is cheap relative to losing a clinic.

Set three distinct clocks:

• A provider/SDK request deadline that genuinely interrupts the underlying call.
• A worker deadline below eight minutes, including cleanup and persistence.
• A platform stop deadline above the worker deadline, leaving time to write terminal state.

Do not assume an outer async timeout actually stops nested SDK retries. Disable or tightly bound provider/framework retry layers and test the deadline against a deliberately stalling endpoint. Compounded SDK, framework, and queue retries can exceed the nominal timeout by several multiples unless each layer is bounded deliberately. The queue visibility timeout must exceed the task’s maximum legitimate runtime plus shutdown margin, or be extended by a lease heartbeat; otherwise two tasks can generate the same note concurrently. Successful encounters should be acknowledged independently.

For A, expose actionable lifecycle data, but don’t dump implementation telemetry into the clinician UI. The API can carry attempt_count, last_activity_at, and sanitized error_class, while the plugin should translate those into:

• Generating note — started 2 min ago
• Delayed by AI service capacity — retrying by 9:17
• Taking longer than usual — you can continue waiting or chart manually
• Note generation failed — transcript is available; Retry / Send to human scribe
• Note ready

Once the charting threshold is crossed, stop showing “batch running: 80%” as the primary truth. Show counts such as 30 ready, 3 generating, 4 delayed, 1 needs attention, and let the clinician open the exact encounter. An aggregate percentage without terminal per-encounter states can remain apparently healthy indefinitely and prevents the user from deciding which chart requires manual action.

I would not push incomplete note sections this week. Streaming partial clinical prose creates a risk that an unfinished or later-revised statement is mistaken for a completed clinical note. It also creates more PHI-bearing persistence, websocket, audit, and retention paths. A simple activity signal—“model is responding”—gives transparency without presenting incomplete clinical content. Token counts belong in operations telemetry, not the clinician display.

B as proposed is actively harmful. A 50-token probe immediately before every real request doubles request count at the worst possible time, and a successful tiny request says little about whether a long generation will finish. During a 9am throttle event, per-job probes consume scarce capacity and can amplify congestion. A four-second timeout followed by fallback can also create duplicate in-flight work if cancellation does not actually terminate the original provider request.

Replace B with a shared, passive circuit breaker and admission controller driven by real request outcomes: timeout rate, 429s, latency, and recent successes by model and region. Honor Retry-After, cap concurrency, add jitter, and reserve capacity separately for clinical notes versus billing. A low-frequency synthetic probe is reasonable only while a circuit is half-open—not once per encounter. Without separate queues or concurrency budgets, autoscaling note workers can continue starving unrelated billing work even after tasks are isolated.

“Poison” classification also needs care. A timeout is not automatically deterministic. Record a fingerprint of the model, prompt-template version, relevant input shape, and error class; after a small bounded number of equivalent failures, route it to DLQ or human handling. Validation/context-limit errors can be terminal immediately, while 429s and transient 5xx responses should be retried within the clinician’s remaining charting window. Once that window is exhausted, call it deferred or needs_attention, not endlessly retrying.

The mitigation I would refuse to ship without is a tested, end-to-end hard wall-clock bound on the actual model call plus durable pre-attempt state. C without that merely delegates timeout handling to SIGKILL; D without that gives you an excellent record of workers hanging forever. I would test four failure points before Thursday: stall before first token, stall mid-stream, provider response followed by process death before DB commit, and queue visibility expiry during generation.

Finally, sanitize everything around failures. Clinic-ops chat should contain encounter references and error classes, not transcript fragments, prompts, generated text, or raw provider error bodies. A cloud BAA does not by itself make every service tier, log sink, websocket path, or support notification appropriate for PHI.

So the week-one decision is: D first, C immediately behind it, A with actionable states, and no per-request B. Add a passive fleet-level breaker, separate workload concurrency, and a stalling-endpoint test. That directly fixes both failures you experienced: invisible encounters and a single prompt consuming an entire batch’s fate.

Ship A + D this week as the non-negotiable core, with a hard per-task wall-clock bound borrowed from C (8 minutes matching clinician tolerance), and treat B as a later experiment only after you have the durable attempt rows and per-encounter visibility. Do not ship full per-encounter isolation (pure C) under Thursday pressure unless your container platform already makes one-task-per-message trivial; the cost and orchestration change is real and secondary to making hangs observable and terminal.

Why this combination

The failure mode is classic: a managed LLM stream hangs, the supervisor kills the task, two encounters in the same batch finished earlier, the rest reappear only after visibility timeout, and the product has no durable row for the hung encounter so the EHR plugin and clinic dashboard stay stuck on aggregate “pending.” Queue visibility-timeout semantics mean an in-flight message that is never deleted or explicitly failed simply becomes visible again for another consumer; without a durable per-message attempt record you cannot distinguish “still running,” “lost to kill,” or “poison.” Nightly reconcile on note_status IS NULL is too late for the clinician’s charting window.

D is mandatory groundwork under time pressure. Insert a note_attempts (or equivalent) row with status='running', encounter_id, attempt_count, leased_at, worker_id, and an idempotency key in the same database transaction that acquires the lock/lease, before any call to the managed LLM endpoint. This is the transactional outbox / pre-attempt pattern: the business intent and the attempt record either both exist or neither does, so a watchdog kill or crash cannot leave a silent hole. On supervisor kill or reaper scan, mark status='lost' (or failed with error_class='timeout_or_kill'). Deterministic timeout shapes and repeated model hangs go to a DLQ with classification rather than infinite re-visibility. This also gives you the place to emit the missing cost/latency metrics that currently vanish on kills. Without D you cannot safely re-drive, cannot poison-classify, and cannot feed A accurate terminal states. Ship the insert + reaper first; everything else sits on top of it.

A is the product fix the clinic actually feels. Enrich GET /api/clinics/{id}/batch-progress (and the existing websocket if it is already carrying scribe events) with per-encounter fields: stt_status, note_status (pending / running / streaming / deferred / failed / lost), token_counts or last chunk size, last_chunk_at, attempt_count, and a short human reason when terminal. Plugin polls at 5 s; optionally push incomplete sections. Aggregate “~80 %” is lying to the dashboard because job_groups only stores totals. The clinician needs to see, for the specific encounter, that note generation is alive versus dead so they can chart manually inside their window instead of waiting forever. Progressive transparency turns the hang from an invisible trust-killer into a recoverable “this one is delayed—chart now or wait for retry.”

Borrow the hard stop from C without full isolation if time is tight: configure the container-platform task (or an inner watchdog around the LLM call) for an 8-minute max runtime that matches stated consumer tolerance, then explicitly write the failure row and notify clinic ops. Full one-task-per-encounter (or small shard) isolation is cleaner for blast radius and concurrency fairness—unrelated billing sync no longer waits behind a hung note batch—but the estimated +18 % compute on peak days and the platform change are not free. Do the isolation next sprint if the hard timeout + D already stops the multi-encounter contamination.

Hidden tensions and direct answers to your questions

Does B become denial-of-capacity at 9 am rush when the managed LLM endpoint throttles? Yes, it can. A 50-token “reply OK” probe every real note request adds load exactly when the endpoint is already slow or returning 429s. Most agent and gateway stacks treat 429 + Retry-After as standard backoff; a probe layer that itself times out at 4 s will mass-fail under throttle and either stampede a fallback model or flood note_deferred for human scribes. At clinic open you risk turning transient capacity pressure into systemic deferral. Pre-flight probes are useful later as a circuit breaker after you have attempt rows and can rate-limit the probes themselves, but they are not the right first patch. Prefer an outer wall-clock timeout on the real call (the GitHub issues on LLM completion hangs show that per-request timeouts plus compounded retries still need a hard backstop that actually cancels) and let D + reaper handle the lost cases.

Does C reduce PHI blast radius enough to justify cost? Partially. Smaller tasks mean less concurrent PHI held in a single long-lived process memory image and a tighter window of exposure if a container is compromised or logged incorrectly. HIPAA still requires the BAA with your cloud provider, encryption in transit (TLS 1.2+) and at rest (AES-256 class), access controls, and audit logging of PHI access for the full retention period; isolation does not relax any of those. The cost justification is more about operational isolation (one hang no longer blocks the batch or unrelated queues) than pure PHI surface. Under Thursday pressure the hard 8-minute stop plus D already shrinks the worst-case clinician wait; full isolation can wait unless your platform makes it free.

Is D mandatory groundwork under time pressure? Yes. Without the pre-attempt row you have no durable signal that an encounter ever entered note generation, the reaper cannot mark lost, the DLQ cannot classify poison prompts, and A has nothing accurate to display. Nightly reconcile remains paperwork. The outbox literature is unambiguous on this: insert the intent record in the same transaction as the lease before the external API call, then drive retries from the durable state with backoff and a hard attempt cap.

What should the dashboard say while a note is failing so the clinician can act? Per encounter, not just the batch: “Note generation running (attempt 1, last activity 47 s ago)” while streaming; “Note delayed — retrying (attempt 2 of 3)” on re-visibility; “Note generation failed / timed out — chart manually or request human scribe” on terminal lost/failed with a one-click “re-queue” or “assign scribe” if your ops flow supports it. Surface last_chunk_at so they can see staleness. Aggregate batch progress should drop the failed encounter out of the “running” count immediately. The goal is that the clinician never sees “Transcribed · Note pending…” forever; they see a clear decision point inside the ~8-minute charting window.

What are we not seeing, and which mitigation would you refuse to ship without?
We are not seeing hang frequency or duration because cost/latency metrics do not emit on watchdog kills—the exact cases that matter. We are not seeing per-encounter terminal state, so autoscaling and concurrency limits keep feeding the same poison shape. We are not seeing whether the hang is model/region specific, prompt-shape specific, or pure network stall. Compounded retries (provider + library + adapter) can still exceed any soft timeout if cancellation is not clean.

I would refuse to ship without D (durable pre-attempt insert + reaper to lost/failed) and without A’s per-encounter status. I would also refuse to ship an LLM path that still lacks a universal wall-clock timeout that actually interrupts a stalled stream—soft timeouts that leave an await blocked until the supervisor SIGKILLs are how you got here. I would refuse pure B as the primary fix this week because of the 9 am capacity risk. Full C is desirable but not the hill to die on before Thursday if the hard timeout + D already contain the blast.

Implement D’s schema and insert path first (reuse your existing lease transaction), wire the reaper to mark lost rows and feed the DLQ, then expose those statuses through A’s progress endpoint and plugin. Add the 8-minute hard stop around the generation task. Instrument the kill path so the next hang emits metrics. That combination restores clinician visibility, stops silent stuck rows, and gives you the ground truth needed to decide on B and full isolation next week without babysitting the Monday re-run.

For this week you need D + C + a minimal slice of A. Defer active B.

Why D is non-negotiable

You have a classic partial-failure: stt_completed committed, clinical_notes row never pre-inserted, worker killed by 15-min supervisor. Recovery is impossible because state lives only in memory.

You need the transactional outbox guarantee: writing the business row and the attempt/outbox row in the same DB transaction prevents "saved in ours, missing in theirs".

Implementation for Thursday:

• note_attempts(id, encounter_id, attempt_no, status, locked_by, locked_at, last_chunk_at, token_count, error_class, idempotency_key) - no PHI, only metadata.
• In one transaction: SELECT ... FOR UPDATE SKIP LOCKED to claim lease + INSERT note_attempts status='running' + INSERT clinical_notes status='pending' if not exists. Idempotency_key = encounter_id:attempt_no unique constraint so racing workers after visibility timeout can't double-create.
• Stream loop: update last_chunk_at, token_count in separate heartbeat table every 5-10s, not the main notes row - avoids write contention.
• Reaper every 2 min: status='running' AND locked_at < now()-10m -> status='lost', release lease. That's what populates missing metrics when watchdog kills - you emit from reaper, not from the dead container.

Retry classification: retry timeouts, network, 429/5xx; 4xx / validation -> DLQ with error_class final. Otherwise the hung prompt shape you saw will be retried forever by the autoscaler.

And you need a real wall-clock cap, not just container supervisor. The search pack shows the prior interim fix was LLM_CALL_TIMEOUT_SECONDS default 120 but native SDK paths were not covered and asyncio.wait_for did NOT cleanly cancel instructor's retry loop - outer wrapper surfaced at ~175s not 60s. You must enforce timeout inside the SDK adapter + a total attempt budget ~6-7m. Otherwise C's 8-min hard stop is the only thing saving you.

C - isolation, yes, even at +18%

Current: one task holds ~38 transcriptions in memory. One hung enc_7f2c blocks N messages until visibility timeout, and billing sync on same namespace blocks on concurrency limits. When a consumer fails to delete before visibility timeout, SQS makes message visible again - so your remaining 35 encounters only reappear after timeout, far past clinician tolerance.

One task per encounter with hard stop 8 min (SIGTERM at 7m, SIGKILL at 8m):

• Blast radius PHI: 38 -> 1 transcript in process memory. For HIPAA that is not just best practice, it reduces audit/ breach scope. That alone justifies +18% for a week patch for 14 clinics.
• Failure is per-encounter terminal, not batch-aggregate. You can write explicit note_failed + ops chat.
• You also fix the visibility timeout anti-pattern: visibility timeout should be hard_stop + 30s, not 15m.

Tradeoff: more cold starts, but you were already paying for autoscaled workers re-hitting same hung prompt.

A - minimal, not full

You need A to make D visible, but not full token streaming this week.

Ship:

GET /batch-progress returns per-encounter: stt_status, note_status[queued|streaming|stalled|retrying|deferred|failed], attempt_count, last_chunk_at_seconds_ago, error_class.

Dashboard wording > raw status:

• 0-2m: "Generating draft... streaming (340 tokens)"

• 45s no chunk: "Note generation stalled - still trying. You can start manual charting; we'll deliver if it arrives"

• 8m hard stop / lost: "Draft unavailable after 8 min - please chart manually [Retry draft]" + WebSocket push

Why not 5s poll for everyone yet: polling all clinics every 5s x 38 encounters will hammer DB. Push lease updates via your existing websocket, but PHI must be encrypted in transit TLS 1.2+ and at rest AES-256 and the vendor must be in BAA scope. Don't put transcript fragments in logs/metrics - token counts only or you create a HIPAA logging violation.

B - will hurt you at 9am

Active 50-token probe per encounter doubles QPS under throttle. When managed endpoint throttles with 429 + Retry-After, frameworks are expected to backoff; adding probe traffic accelerates hitting the burst bucket.

And your failure was mid-stream hang, not connect failure. A probe that returns OK in 1s does not predict a hang at token 800.

Refuse active B. If you want circuit breaking, make it passive this week:

• Shared counter in Redis per model/region: sliding window 2 min. 5 timeouts/5xx -> open circuit 60s, return note_deferred immediately without calling LLM. Half-open with one probe, not N probes.
• No fallback that sends PHI to a non-HIPAA-eligible tier.

What you're not seeing

1. No metrics on watchdog kills = your worst case invisible. Reaper must emit note_attempt{error_class=watchdog_kill}.
2. Single queue namespace for notes + billing sync. Split or at least separate concurrency pools.
3. Prompt shape poison: that encounter that hung - is it reproducible? Needs poison-prompt fingerprint -> DLQ, not infinite retry.
4. Nightly reconcile note_status IS NULL AND stt_completed_at < now()-2h misses D - you need reaper, not nightly.

What I refuse to ship without

D + 8-min enforceable cap + explicit terminal state.

Without D you can make failure visible (A) but you still have forever-pending EHR rows and at-least-once duplicates after visibility timeout. Without hard cap, B is theater.

If time is tight: D + C (shard size 1, 8-min kill) + minimal A (per-encounter status + Dashboard messages above) + passive breaker. Put B active probing on next sprint with proper load testing.

I’m firmly for full C on Thursday. Given the stipulation that it is a one-engineer-day configuration change, deferring it would preserve the exact failure coupling that hurt the clinic. The honest sentence for the 18% is:

The extra compute buys encounter-level failure isolation: one hung note can consume at most one task and one notes concurrency slot, rather than delaying the rest of the clinic’s batch.

I would not sell this as meaningful containment of a security breach. With shared IAM, logging, and network policy, one-task-per-encounter reduces co-resident PHI in process memory but does not materially narrow what a compromised workload can access or exfiltrate. @Muse’s memory-residency point is directionally real, but it is not the justification I would put before compliance or the clinic.

Thursday ship list

1. Durable generation and attempt records. In the same short transaction that claims the logical note version:

   • Upsert note_generations(encounter_id, note_version, status, active_attempt_id, lease_epoch, lease_expires_at).
   • Lock that row.
   • Insert note_attempts(..., status='running').
   • Point active_attempt_id at the new attempt, increment lease_epoch, and commit.
   • Never hold this transaction across inference.

2. Use an encounter-level fenced claim, not encounter_id:attempt_no alone. The invariant is one active owner for (encounter_id, note_version), enforced by the locked note_generations row. A claimant may proceed only if there is no active unexpired lease. If the lease expired, it marks the prior attempt lost, increments lease_epoch, and installs the replacement attempt atomically.

   Every heartbeat and final write must be conditional:

   UPDATE note_generations
      SET status = 'ready', active_attempt_id = NULL
    WHERE id = :generation_id
      AND active_attempt_id = :attempt_id
      AND lease_epoch = :claimed_epoch;
   

   A zero-row update means the worker has lost ownership and must discard its result. This fencing check prevents a stale worker from publishing after a replacement worker has acquired the encounter, even if cancellation of the stale provider call failed. Keep a unique constraint on (generation_id, lease_epoch) for attempts and (encounter_id, note_version) for the logical generation.

   The external idempotency key, if the managed endpoint offers well-defined idempotency, should be stable for the logical generation, not derived merely from a locally guessed attempt number. That limits duplicate provider work across retries; the fencing token remains authoritative for which result the product accepts.

3. One encounter per task, with workload isolation. Give clinical notes a concurrency budget separate from billing and launch each encounter as its own one-off task. A poison encounter may occupy one notes slot, but it cannot retain a batch of unrelated queue messages. Ensure the launcher itself cannot monopolize capacity by repeatedly dispatching the same generation while its lease is active.

4. Enforce the three clocks.

   • Provider/adapter deadline: about six minutes, with SDK retries included in that total.
   • Worker deadline: about seven minutes, allowing persistence and cleanup.
   • Platform hard stop: eight minutes.
   • Queue visibility: longer than the platform bound plus launch/shutdown margin, or extended by a heartbeat.

   Test that the provider deadline stops network consumption rather than merely abandoning a local await. Test stalls before the first token and midway through the stream. Layered retries inside the SDK, adapter, and worker can otherwise make a nominal request timeout substantially exceed its intended wall-clock budget.

5. Add the reaper and terminal accounting. The reaper uses lease_expires_at, not merely an old locked_at, and conditionally marks the matching active attempt lost. It emits duration and failure metrics, updates batch counts, and either schedules a bounded retry or marks the generation failed/deferred. Graceful SIGTERM handling may accelerate this, but correctness must not depend on it.

6. Ship minimal A. Return per-encounter lifecycle status, attempt count, last activity time, expected next action, and a sanitized error class. Keep token counts and provider details in operational telemetry. Use the existing websocket for state changes if convenient, with polling as fallback; do not stream partial clinical prose.

7. Ship the passive breaker. Track outcomes by model and region, but distinguish 429 throttling, transport timeout, mid-stream stall, and provider 5xx. Honor Retry-After; use load-aware thresholds rather than a fixed “five failures” rule at every traffic level. Half-open with one controlled request. Do not let circuit opening create an unbounded stampede to a fallback deployment.

8. Make manual charting a product state, not just dashboard copy. “Chart manually” should atomically mark the generation manual_charting or delivery_suppressed, increment the fencing epoch, and prevent a late model result from being posted into the chart. Without this transition, better transparency can create a new race in which the clinician charts manually and a delayed generated note subsequently appears.

Final poison rule

Fingerprint the exact generation conditions with a keyed HMAC rather than storing prompt text:

HMAC(
  canonical rendered prompt bytes
  || prompt_template_version
  || note_schema_version
  || model/deployment version
  || generation parameters
)

Store separately the failure stage and class: validation, context_limit, before_first_token_timeout, mid_stream_stall, malformed_output, and so forth. Do not include raw excerpts in metrics, DLQ labels, or clinic-ops chat.

My rule would be:

• One failure for deterministic local or provider validation errors—unsupported request, context limit after deterministic tokenization, schema impossibility—goes terminal immediately.
• Two equivalent failures with the same fingerprint, model/deployment, and failure stage, on separate leased attempts, mark that model-specific generation poison only if the regional breaker does not indicate broad degradation.
• 429, 5xx, network failures, and timeouts during a regional incident never establish prompt poison by themselves; they consume the retry/time budget and then become deferred_capacity.
• If an approved fallback model exists, its poison history is separate. Failure on model A does not prove the prompt will fail on model B.

Conditioning poison classification on fleet health is important because two identical mid-stream stalls during a provider incident are evidence of provider degradation, not strong evidence that the encounter content is intrinsically poisonous.

Clinician-visible state machine

The clinician should see these states, not internal terms such as leases, DLQs, or lost:

• Transcript ready · Note queued
• Generating note · started 2 minutes ago
• Generation is taking longer than usual · still trying
  Actions: Keep waiting or Chart manually
• AI service is busy · retry scheduled by 9:17
  Actions: Chart manually or Send to human scribe
• Draft unavailable after 8 minutes
  Actions: Retry draft, Chart manually, or Send to human scribe
• Charting manually · automatic draft delivery stopped
• Note ready

The batch header should say, for example, 30 ready · 3 generating · 4 delayed · 1 needs action, with “needs action” visually dominant. A failed encounter must leave the running denominator immediately.

Refusal line

I would refuse to ship Thursday without D’s durable fenced claim and reaper, a tested provider-level cancellation bound, an explicit clinician-visible terminal state, and one-encounter-per-task isolation. Under the moderator’s implementation stipulations, all four address demonstrated production failures rather than speculative hardening.

I now own full one-encounter-per-task (C) for the Thursday cut. The prior caution against pure C under time pressure is withdrawn: with the platform already running one-off tasks per message for the billing exporter, the change is a one-engineer-day templating exercise, not an orchestration project. That fact removes the main cost of isolation this week.

The one honest sentence that justifies the pre-approved +18 % compute, given that every task still shares one IAM role, one log sink, and one network policy: it buys failure-domain isolation so that a single hung generation can no longer strand an entire clinic batch behind one worker, force the remaining messages to wait out visibility timeout past the charting window, or starve unrelated billing workers on the same concurrency pool. Process-memory reduction (38 transcripts to 1) is a secondary hygiene win; it does not meaningfully shrink authorization or infrastructure blast radius under the shared controls, so I will not defend the +18 % on HIPAA audit-scope grounds alone.

@GPT’s three-clock layering and explicit state machine are the right backbone; @Muse’s reaper-every-2 min and heartbeat table are the right operational details. Both correctly kill active per-request B and partial clinical prose. I build on those and close the two open design threads.

Atomic encounter-level claim (idempotency)

encounter_id:attempt_no alone is insufficient: two workers that both observe a lost lease can race, each create a higher attempt number, and both call the model. The durable claim must be at the logical note / version, not merely the attempt.

In a single short transaction:

1. SELECT \ldots FOR UPDATE SKIP LOCKED on the encounter (or a dedicated note_leases row keyed by encounter_id + note_version).
2. Assert that no active generation exists (note_status not in generating, retry_scheduled with unexpired lease).
3. Upsert clinical_notes to note_status = 'generating', set active_generation_id = new_uuid(), lease_expires_at = now() + interval '8 minutes'.
4. Insert note_attempts(status = 'running', attempt_no = prior + 1, generation_id = that same uuid, \ldots) with a uniqueness constraint on (encounter_id, generation_id).
5. Commit.

Only then open the provider stream. Heartbeats update a separate lightweight row (last_chunk_at, token counters) every 5–10 s so the main note row is not contended. On success, the same worker (or a completion handler) transitions the note to ready and deletes the queue message. The reaper owns the running → lost transition when lease_expires_at passes; the killed process is never trusted to write its own terminal state. A stable provider-side idempotency key, when available, is set to the generation_id so a late-arriving response after a reaper mark cannot create a second note.

This gives at-most-one active generation per logical note while still allowing ordered retries after a true lost or retry_scheduled decision.

Poison classification

A timeout or mid-stream hang is not automatically poison. Fingerprint = hash(model_id + prompt_template_version + input_shape_bucket) where the shape bucket is non-PHI (audio-duration quartile, transcript-token quartile, specialty code, encounter-type). After two equivalent failures (same fingerprint + error_class in {timeout, stall, 5xx}) inside a 24-hour window for that clinic’s model/region, the third occurrence is classified poison: mark the attempt failed with error_class = 'poison_shape', move the queue message to the DLQ, set the note to deferred_to_human, and surface the actionable state. Validation / context-limit / 4xx errors are terminal on first occurrence. 429s and pure network blips stay in the normal retry budget until the clinician charting window is exhausted, then become needs_attention. The reaper and the passive circuit breaker both read the same fingerprint counters so a poison shape does not keep consuming capacity.

Thursday ship list (numbered)

1. D first: schema for note_attempts (metadata only, no PHI) + clinical_notes lifecycle columns; atomic claim transaction above; reaper every 2 min that marks expired leases lost, emits the missing note_attempt{error_class=watchdog_kill|timeout} metrics, and releases the lease. Never hold the DB transaction across the model call.
2. Three clocks + actual cancellation: provider/SDK deadline that interrupts the underlying stream (tested against a deliberately stalling endpoint; no reliance on outer wait_for alone); worker deadline ≤ 7 min including cleanup and terminal write; platform stop at 8 min. Bound or disable compounded library/framework retries. Visibility timeout = hard-stop + 30 s (or lease heartbeat).
3. C: one container-platform task per encounter (config template already proven by the billing exporter). Hard stop as above. Successful encounters acked independently.
4. A (focused): enrich batch-progress and the existing websocket with per-encounter lifecycle only. No partial clinical prose, no token counts in the clinician UI.
5. Passive breaker only: shared sliding-window counters (timeouts, 429s, latency) per model/region; open circuit → immediate note_deferred without a new call; half-open with a single synthetic probe; separate concurrency budgets for notes versus billing so autoscaling notes cannot starve the other workload. Honor Retry-After.
6. Sanitization and ops: clinic-ops notifications carry encounter id + error_class only; no transcripts, prompts, or raw provider bodies. Confirm BAA coverage, TLS 1.2+, AES-256 at rest for every new path.

Dashboard state machine (clinician view)

• Generating note — started X min ago (while lease live and chunks arriving).
• Taking longer than usual — you can continue waiting or chart manually (no chunk > 45 s, still inside window).
• Delayed by AI service capacity — next retry by HH:MM (circuit open or retry_scheduled).
• Note generation failed — transcript available; [Retry draft] [Send to human scribe] (lost / poison / terminal after window).
• Note ready.

Batch header becomes counts, never a single healthy-looking percentage: “28 ready · 4 generating · 3 delayed · 1 needs attention.” Once any encounter crosses the charting threshold it leaves the “running” bucket so the UI cannot lie.

What I refuse to ship without (surviving positions)

• D + reaper that writes terminal state after kills — without it every other control still leaves forever-pending EHR rows and invisible metrics.
• A wall-clock bound that actually cancels the provider stream, proven on a stalling endpoint — without it the 8-minute platform stop is just SIGKILL theater and C merely moves the silent hang.
• Full C this week now that it is a config change — without isolation a single poison shape still contaminates the batch and the shared concurrency pool.
• Actionable per-encounter states with the plain-language wording above — without them the clinician still cannot decide inside the charting window.
• Passive breaker + separate note/billing budgets — without them 9 a.m. capacity pressure or a hung shape still cascades.

That cut restores visibility, makes hangs terminal and observable, contains blast radius at the failure-domain level, and gives the Monday re-run a system that fails loudly instead of silently. Everything else (active probing, partial streaming, richer telemetry in the UI) waits.

Given the stipulations, I flip on C: ship full one-encounter-per-task Thursday.

In round 1 I held back from pure C unless one-task-per-message was already trivial - worried about Thursday orchestration risk. With the billing exporter already running one-off tasks per message and the change estimated at one engineer-day, that risk is gone. And my R1 line that C cuts PHI audit/breach scope by going 38->1 in memory is not defensible under stipulation 3 - shared IAM role, shared log sink, shared network policy means isolation changes none of the HIPAA authorization footprint this week.

The honest sentence that justifies +18%: +18% buys strict failure-domain isolation - one poisoned prompt can delay and kill only itself, not strand 37 unrelated charts behind its visibility timeout and autoscaler slot.

Thursday cut

1. D - durable pre-attempt + atomic claim, committed before LLM call

• In one transaction: SELECT clinical_notes WHERE encounter_id=$1 FOR UPDATE SKIP LOCKED. If status IN ('generating','retry_scheduled') AND lease_expires_at > now() -> skip, already active.
• Else: UPDATE clinical_notes SET status='generating', generation_id=$gen, lease_expires_at=now()+8m, version=version+1 + INSERT note_attempts(generation_id, encounter_id, attempt_no, status='running', idempotency_key=generation_id, worker_id, leased_at)
• Commit, then call provider. Never hold tx across inference.

Fixes @GPT's idempotency objection: encounter_id:attempt_no alone does not prevent two workers under two different attempt numbers from generating concurrently after visibility timeout. You need the encounter-level claim.

• generation_id is the logical note/version idempotency key. Unique partial index: CREATE UNIQUE INDEX ON clinical_notes(encounter_id) WHERE status NOT IN ('ready','failed','deferred','deferred_to_human') - only one active generation per encounter.
• Provider idempotency header uses stable generation_id, not attempt_no, so even racing workers dedup. note_attempts has UNIQUE(generation_id, attempt_no) plus FK to active claim.

2. C - one task per encounter, not shards Task template = billing exporter. Visibility timeout = worker max + 30s, not shared batch. Independent ack means two encounters in batch no longer share fate with enc_7f2c.

3. Three clocks that actually cancel Outer asyncio.wait_for does not cleanly cancel instructor/litellm retry loops - validated to surface at ~175s for a 60s deadline.

• SDK deadline inside adapter that interrupts underlying socket - tested against stalling endpoint (stall before first token + stall mid-stream)
• Worker deadline 7m30s soft (stop streaming, write failure) / 8m hard SIGKILL
• Platform stop 9m30s, leaving time to persist terminal state
• Disable/tightly bound provider + framework retries. Compounded retries can exceed nominal timeout by multiples unless each layer bounded deliberately

4. Reaper + metrics emission Reaper every 60s: status='running' AND lease_expires_at < now() -> status='lost', error_class=timeout_or_kill|watchdog. Reaper emits cost/latency{error_class} that watchdog kills currently hide. A process killed by the platform cannot be relied upon to record its own failure - termination handlers are best-effort.

5. Passive breaker + workload isolation, not active B Active 50-token probe per request doubles QPS under throttle and mass-fails at 9am, turning transient 429s into systemic deferral. Ship passive:

• Shared Redis counters per model/region: timeout rate, 429s (honor Retry-After), p95 latency
• 5 failures / 2min -> open 60s, return deferred without calling LLM, half-open with one probe
• Separate concurrency budgets / queue namespaces for notes vs billing sync so autoscaled note workers cannot starve billing.

6. A minimal - actionable, not telemetry dump API carries attempt_count, error_class, last_activity_at but clinician UI shows plain language, no token counts, no partial prose.

Poison classification - final shape

Fingerprint must contain no PHI: {model_id, region, prompt_template_version, error_class[timeout_before_first_token|mid_stream_stall|context_limit|validation|5xx|429], transcript_duration_bucket[0-5m,5-15m,15-30m], stt_confidence_bucket} - not transcript text, not encounter_id.

• validation, context_limit -> terminal immediate -> failed -> DLQ human scribe
• 5xx, 429, timeout_* -> retry up to 2 equivalent fingerprint failures per encounter; 3rd equivalent -> deferred_to_human + DLQ with fingerprint hash. After 2, same shape across different encounters within 1h -> circuit opens for that shape.

Dashboard state machine - what clinician sees

queued -> generating -> ready generating -> retry_scheduled -> generating generating -> lost/failed -> needs_attention | deferred_to_human

UI copy:

• Queued - In line to generate
• Generating - started ~1m ago - live since last chunk <45s
• Taking longer than usual - you can chart manually, we'll deliver if it arrives - no chunk >45s, still retrying, Attempt 2
• Delayed by AI capacity - retrying by 9:17 - breaker/429 path
• Note failed - transcript available - Retry draft / Send to human scribe - terminal failed/lost
• Note ready - done

Batch header stops showing "80% running". Show 30 ready, 3 generating, 2 delayed, 1 needs attention - per-encounter terminal state is primary truth. Aggregate lying is how you got invisible pending.

What I refuse to ship without

• D without holding tx across inference: Without durable pre-attempt row + atomic generation claim, hangs recreate invisible null-row forever.
• End-to-end wall-clock bound that actually interrupts the LLM call, tested vs stalling endpoint: Otherwise D just records hangs.
• No active per-request B: Passive breaker only - active probe is capacity self-DoS.
• Per-encounter terminal states in UI (A) + separate concurrency for billing: Otherwise single poison prompt still blocks batch and billing.
• No PHI in fingerprints, logs, or ops chats: BAA does not make every sink eligible; token counts stay in ops telemetry only.

The convergence is sound, but I would stop calling the retry rule “poison detection.” We cannot infer that encounter content is intrinsically poisonous from two hangs. What we can safely implement is exact-request repeated-failure quarantine.

Final poison/quarantine rule

Fingerprint the canonical request with a keyed HMAC:

request_fingerprint = HMAC-SHA256(
  canonical_prompt_bytes
  || model_and_deployment_version
  || prompt_template_version
  || note_schema_and_tool_schema_version
  || generation_parameters
  || streaming_mode
)

Canonicalization removes request IDs, timestamps, and other non-semantic fields. Store the HMAC, never the rendered prompt. Treat even the HMAC as sensitive operational metadata, rotate and restrict its key, and never put it in clinic-facing messages.

Failure equivalence is:

request_fingerprint
+ model/deployment
+ failure_stage
+ normalized_error_class

The shipped policy is:

1. Deterministic local validation, context-limit, unsupported-parameter, or schema errors are terminal after one occurrence.
2. A before-first-token timeout, mid-stream stall, or malformed-output failure gets one retry, provided sufficient charting time remains.
3. If the same canonical request fails at the same stage on two fresh tasks/connections while that model/region’s fleet breaker is healthy, stop automatic retries and mark it repeated_request_failure.
4. Generic 429, 5xx, DNS, connection, and regional timeout events never count toward request quarantine. They feed the provider breaker and become deferred_capacity when the charting budget expires.
5. If the fleet is degraded during either attempt, discard that attempt from the request-specific counter. Do not retroactively call it poison after recovery.
6. Ship no shape-level or cross-encounter circuit Thursday. @Grok’s and @Muse’s duration/confidence buckets are too coarse, and n=2 at this clinic volume cannot distinguish a bad shape from coincidence or provider degradation. A two-sample bucket circuit would have an unacceptably unstable false-positive rate without substantial baseline volume and per-bucket failure-rate estimates.

Two equivalent failures do not prove causality; they are simply the point at which another automatic attempt has worse expected value than returning control to the clinician. The clinician sees:

Draft unavailable for this encounter — transcript is ready.
Chart manually · Send to human scribe · Try approved alternate model

Do not offer an immediate retry against the identical deployment unless an operator clears the quarantine or the request/model version changes. Selecting “Chart manually” durably transitions to manual_charting, revokes the active generation, increments its fencing epoch, and suppresses late delivery.

Fencing and uniqueness: ship both

These are complementary, not redundant.

Use a durable note_generations row for each logical (encounter_id, note_type, version), with active_attempt_id, lease_epoch, lease_expires_at, and lifecycle state. Acquisition should be one atomic transaction—preferably an INSERT … ON CONFLICT … DO UPDATE … WHERE lease_expires_at < now() RETURNING … or an equivalent locked-row transition—rather than “observe, then insert.”

Every heartbeat, failure transition, and completion must require both the attempt ID and epoch:

UPDATE note_generations
SET status = 'ready',
    active_attempt_id = NULL,
    lease_expires_at = NULL
WHERE id = :generation_id
  AND active_attempt_id = :attempt_id
  AND lease_epoch = :epoch;

Zero affected rows means ownership was lost; the worker must discard its result.

The lease epoch catches stale ownership: worker A stalls, the reaper permits worker B to take over, and A later returns. Without fencing, A can still publish after B owns the generation.

The partial unique index catches competing logical generations caused by application bugs, duplicate dispatch, or two code paths selecting different version/attempt IDs:

CREATE UNIQUE INDEX one_active_note_generation
ON note_generations (encounter_id, note_type)
WHERE retired_at IS NULL;

Also retain ordinary uniqueness on (encounter_id, note_type, version) and (generation_id, lease_epoch). The exact predicate should represent durable active-generation membership, not a long negative list of statuses that can silently become wrong when a new status is added. Using status NOT IN (...) as the uniqueness predicate is vulnerable to future states accidentally being treated as active.

Provider-idempotency dependency, in one line:

If provider idempotency is weaker than documented, we may pay for duplicate inference and receive duplicate streams, but encounter-level fencing still prevents a stale or duplicate worker from publishing a second local note.

Local correctness must not depend on provider behavior.

One clock ladder

Commit to this ladder:

1. Provider/adapter wall-clock deadline: 6 minutes. This includes all internal retries and must terminate the socket/stream, not merely cancel the outer await.
2. Worker soft deadline: 7 minutes. Stop new work, classify the attempt, write terminal state, and acknowledge or release appropriately.
3. Platform hard stop: 8 minutes. Kill anything that ignored the first two clocks.
4. Queue visibility: 9 minutes, or heartbeat-extension capped at 9 minutes. It must outlive the platform stop and cleanup margin.
5. Lease expiry: 7 minutes 30 seconds, renewed only by the owning attempt and never beyond the 8-minute platform budget.

Test stalls before the first token, midway through streaming, and during SDK retry backoff. Also test a provider response followed by worker death before the local commit. Those cases exercise different cancellation and ownership paths and will not necessarily be covered by one generic timeout test.

Thursday ship list

1. Durable note_generations and note_attempts, created and claimed before inference.
2. Encounter-level atomic claim, lease epoch, conditional fenced writes, and the active-generation unique index.
3. Reaper that marks expired attempts lost, emits metrics, and decides bounded retry versus terminal handling.
4. One encounter per one-off task.
5. The 6/7/8/9-minute clock ladder, tested against a stalling endpoint.
6. Separate concurrency budgets for clinical notes and billing.
7. Passive model/region breaker honoring Retry-After; no per-request probe.
8. Exact-request repeated-failure quarantine; no shape circuit.
9. Per-encounter clinician states through the progress API and websocket, with no partial note prose or token telemetry in the UI.
10. Durable manual_charting and delivery_suppressed transitions so late drafts cannot race clinician action.

The honest +18% sentence remains:

The additional compute buys failure-domain isolation: one hung encounter can occupy at most one task and one notes slot instead of delaying the clinic’s remaining charts.

It does not buy a materially smaller authorization or infrastructure blast radius under shared IAM, network policy, and logging.

Clinician-visible state machine

• Transcript ready · Note queued
• Generating note · started 2 minutes ago
• Taking longer than usual · still trying
  Actions: Keep waiting · Chart manually
• AI service busy · retry scheduled by 9:17
  Actions: Chart manually · Send to human scribe
• Draft unavailable for this encounter · transcript ready
  Actions: Chart manually · Send to human scribe · Try approved alternate model
• Charting manually · automatic draft delivery stopped
• Note ready

The header reports counts: 30 ready · 3 generating · 4 delayed · 1 needs action. “Needs action” must not remain in the running denominator.

Attack on our own design

We have made failure loud, and loudness has costs.

First, human fallback can become the next overloaded queue. A provider incident at 9am could turn hundreds of invisible model delays into simultaneous banners, ops messages, and human-scribe requests. Without deduplicated incident notifications, rate-limited ops alerts, and visible human-scribe capacity, progressive transparency can create alert fatigue and promise a fallback that cannot meet demand. Clinic ops should receive one incident aggregate plus encounter drill-down, not one chat message per failed attempt.

Second, status churn can destroy trust almost as effectively as silence. An encounter that flips every few seconds among generating, stalled, retrying, and generating will look unstable. Apply minimum display durations and expose only meaningful transitions; keep heartbeat details internal.

Third, the reaper can amplify a database or heartbeat outage. If workers are healthy but cannot renew leases, the reaper will authorize replacements, increasing provider spend and load. Fencing prevents duplicate publication, but it does not prevent duplicate inference. Pause takeover when the lease store itself is unhealthy, and cap replacement concurrency.

Fourth, one-task-per-encounter moves pressure to task launch, database connections, and cold starts. At rush traffic, task creation and per-task connection initialization can become a new bottleneck even though inference failures are isolated. Put a launch admission limit in front of the task platform and use a connection proxy/pool rather than letting every task open unconstrained database sessions.

Fifth, manual charting introduces a clinical race. If clinician intent is merely UI state rather than a fenced server transition, a late generated draft can still land after manual documentation. That is why manual_charting must revoke ownership, not just hide a spinner.

Finally, the passive breaker itself can cause a deferral cliff. Opening it protects capacity but sends every admitted encounter directly to delayed or human handling. Use hysteresis, bounded half-open traffic, and a clinic-visible incident state rather than pretending each encounter independently failed.

Concession audit

• I began with D + C + focused A and rejection of active B. That position survived.
• I initially treated reduced PHI co-residency as a secondary benefit of C. Under the shared-control stipulation, I sharpened that: it is hygiene, not a defensible reduction in this week’s authorization or breach blast radius.
• I moved from a generic uniqueness/idempotency recommendation to requiring a monotonically fenced encounter-level claim. The standing race objection justified that move.
• I moved from loosely described “deterministic model/timeouts” poison handling to exact-request quarantine conditioned on fleet health. The objection that shape buckets conflate unrelated encounters is decisive.
• I converted the three abstract clocks into one 6/7/8/9-minute ladder.
• I added a durable manual-charting transition after recognizing that transparency creates a late-delivery race.

If Thursday vanished, I would take back the name and rigidity of the two-failure quarantine threshold. I would first run exact-request replay experiments, measure deployment-specific recurrence, and derive thresholds from remaining charting time and observed retry value. For Thursday, two attempts are a reasonable operational budget, not a scientific determination that the prompt is poison.

Verdict

GO for Thursday, provided all of these gates pass:

• durable pre-attempt creation and atomic fenced claim;
• reaper-tested recovery after ungraceful kill;
• demonstrated underlying-call cancellation against both pre-token and mid-stream stalls;
• one encounter per task with separate note/billing concurrency;
• clinician-visible terminal state and durable manual-chart suppression;
• no active pre-flight probe and no shape-level poison circuit;
• no PHI-bearing prompt, transcript, raw provider error, or partial draft in metrics or clinic-ops notifications.

My preserved dissent is narrow: do not present “two equivalent failures” as proof of poisonous content. It is a bounded retry decision made to protect the clinician’s remaining charting window.

I ship the following poison rule on Thursday. Fingerprint is a keyed HMAC over prompt_template_version || note_schema_version || model_deployment_id || generation_parameters || non-PHI shape vector, where the shape vector is only the coarsest safe buckets (audio-duration quartile and transcript-token quartile). No encounter identifiers, no transcript text, no raw excerpts ever enter the fingerprint, metrics labels, DLQ, or ops chat. Failure stage is stored separately: validation, context_limit, before_first_token, mid_stream_stall, malformed_output, 429, 5xx, transport.

Counts and conditioning: deterministic validation or context_limit after local tokenization goes terminal on the first attempt and becomes failed with clinician actions Retry draft / Send to human scribe. For stall or timeout stages, require three equivalent failures (same fingerprint + same stage) on separately leased attempts for that same encounter, and only while the regional passive breaker reports healthy success rate (above 85 % successful completions in the trailing 10 minutes for that model/region). If the breaker is already open or half-open, the same stalls consume the remaining charting-window budget and surface as deferred_capacity without ever setting the poison bit. Cross-encounter shape circuits are not used at n=2; at clinic volume a shape must appear as the third failure on at least five distinct encounters inside one hour and the regional breaker must still read healthy before any shape-level open is allowed. When the rule fires the clinician sees “Draft unavailable — content shape needs review” with the same Retry / Send-to-scribe actions; the generation is marked deferred_to_human and the queue message goes to the DLQ with only the fingerprint hash. This keeps 5xx and regional degradation out of the poison path, avoids quarantining healthy long notes via coarse buckets, and raises the cross-encounter bar high enough that a single bad morning cannot false-positive the clinic.

@GPT’s fleet-health gate and separate fallback-model history are correct and adopted; @Muse’s stage separation is correct and adopted. The earlier n=2 proposals, including mine, were too aggressive for the observed volume.

On the two closes: ship both the partial unique index and lease_epoch fencing. The partial unique index
CREATE UNIQUE INDEX ON clinical_notes(encounter_id) WHERE status NOT IN ('ready','failed','deferred','deferred_to_human','manual_charting')
is cheap DB-level insurance that prevents two active generations from ever being inserted even if application code races. Lease_epoch (incremented on every successful claim) plus the conditional final write
UPDATE \ldots SET status='ready' \ldots WHERE generation_id=:g AND active_attempt_id=:a AND lease_epoch=:e
is the ownership fence: a zero-row update forces the stale worker to discard its result. They are not redundant; the index catches insert races, the epoch catches post-claim publication races after a reaper or manual-chart transition has already moved ownership. Provider idempotency dependency in one line: if the managed endpoint’s idempotency is weaker than documented we may pay duplicate token cost, but the lease_epoch fence still guarantees only one result is ever accepted into the chart.

One clock ladder, numbers committed:

• Provider/SDK deadline: 5 minutes 30 seconds, measured from the first byte of the request, with all library and adapter retries forced inside that budget and proven to close the socket (not merely abandon a local await).
• Worker soft deadline: 7 minutes (stop stream, write terminal state under the fence, ack or release).
• Platform hard stop: 8 minutes.
• Visibility timeout: 8 minutes 45 seconds or extended by lease heartbeat.

Outer asyncio.wait_for alone is known not to cancel nested instructor/litellm loops cleanly; the stalling-endpoint test (before-first-token and mid-stream) is a hard gate before any task template ships.

I also adopt @GPT’s requirement that “Chart manually” is a durable product state: the same short transaction that records the clinician action sets status='manual_charting', increments lease_epoch, and suppresses delivery of any late provider result. Without that fence the improved transparency creates a new race in which a delayed draft overwrites or duplicates a manual chart.

What still fails in the D+C+A design we converged on

We have made every hang loud, but loudness has costs none of us named. Every attempt now writes a claim row, heartbeats, and a reaper transition; at peak clinic open that is a measurable write amplification on the notes tables and a larger surface for lock contention if the partial index or epoch updates ever contend under a thundering herd of re-queues. One-task-per-encounter removes batch coupling yet multiplies cold-start and container-churn cost exactly when the autoscaler is already spinning; the +18 % is pre-approved for isolation, not for the secondary effect of more short-lived tasks thrashing the image cache. Clinician-visible “needs attention” states, if they fire more than a handful of times in a morning, produce alert fatigue and increase human-scribe load on the very Monday re-run we are trying to protect. The design also assumes the reaper itself is highly available; if the reaper process or its DB connection is partitioned, we re-create silent running rows until the next successful scan—exactly the original invisible-null-row failure, only now with more metadata. Finally, fencing and terminal states make failure obvious to the product, but they do not yet address STT quality feeding systematically bad prompts; a clean speech-to-text result can still be a poison shape that we will now correctly isolate, yet the root content problem remains for the human scribe queue.

Concession audit

• I moved from “defer full C unless the platform change is trivial” to “ship full C Thursday” once the billing-exporter precedent and one-engineer-day estimate were stipulated; that was the correct shift and I keep it.
• I retracted the claim that 38\to1 memory residency materially shrinks HIPAA audit/breach scope under shared IAM, log sink, and network policy; the honest justification is only failure-domain isolation.
• I dropped partial clinical-prose pushes after the clinical-safety objection.
• I raised my original poison threshold and added the strong fleet-health gate after the moderator challenges.
• I adopted lease_epoch fencing and the manual_charting product state that I had treated as UI copy only.
• The single move I would take back if Thursday vanished is the early under-weighting of isolation cost: I spent too long treating C as optional scaffolding rather than the direct fix for the batch-stranding failure the clinic actually experienced.

Verdict

GO for Thursday provided the refuse-to-ship conditions hold: durable fenced claim + reaper that writes terminal state after kills (D), provider-level cancellation proven on a stalling endpoint with the 5:30 / 7 / 8 clock ladder, full one-encounter-per-task isolation (C), actionable per-encounter states including the manual_charting fence (A), passive breaker with separate notes/billing concurrency budgets, and zero PHI in fingerprints, logs, or ops notifications.

Dissent I preserve: the stalling-endpoint cancellation test remains a hard gate, not a nice-to-have; without it the rest of the design only records hangs more elaborately.

With the stipulations I withdraw R1 caution and now ship full C. The billing-exporter precedent makes it a config templating change, not an orchestration project. And I withdraw my R1 PHI blast radius line - with shared IAM role, shared log sink, shared network policy, one-task-per-encounter does not materially narrow what a compromised workload can access or exfiltrate. It reduces co-resident memory, but that's hygiene, not justification.

Honest sentence for +18%:

+18% buys encounter-level failure isolation: one hung generation can consume at most one task and one notes concurrency slot, rather than delaying 37 unrelated charts behind its visibility timeout and shared concurrency.

The one live fight: poison rule I would actually ship

@GPT's HMAC-of-prompt is right, shape-buckets are wrong for blocking. Shape buckets quarantine healthy encounters. 5xx never proves content is poison.

Final rule:

• Fingerprint: Primary = HMAC_SHA256(canonical_rendered_prompt || template_version || schema_version || model_id || gen_params) - exact content, no raw PHI stored. Secondary is non-PHI shape bucket (duration quartile, token quartile) for dashboards only, not for blocking.

• Deterministic terminal: validation, context_limit, unsupported -> first occurrence terminal immediately. failed -> deferred_to_human. Does not count toward cross-encounter counter.

• Potentially content-triggered: before_first_token_timeout, mid_stream_stall, malformed_output

  Condition to even consider poison: fleet-health gate healthy. Check passive breaker per model/region rolling 5m: if timeout+5xx+429 rate > 15% or breaker open -> mark deferred_capacity, do NOT increment poison counters. This is how we stop conflating provider degradation with prompt poison.

  Then: same exact HMAC on the SAME encounter_id failing twice on separate leased attempts, fleet healthy both times -> that encounter model-specific poison. Terminal deferred_to_human, DLQ with fingerprint only, clinician sees "Note unavailable - needs human scribe."

Cross-encounter shape circuit: disabled for Thursday cut. At 14 clinics volume, n=2 same shape across different encounters false-positives. If you keep it later, n=2 is only defensible for exact HMAC on same encounter_id. For shape-bucket, require n>=10 distinct encounters, same model/template/error_stage, 24h, fleet healthy, plus manual flag or explicit allowlist - because you need enough base rate to rule out capacity correlation.

Clinician outcome when fires: never "poison." It's "Draft unavailable after 8 min - Retry draft / Send to human scribe" for first two, then "Note needs human scribe - this draft shape failed twice."

Two closes

1. Lease epoch + partial unique index - need both, different failures:

• Partial unique: CREATE UNIQUE INDEX ON note_generations(encounter_id) WHERE status NOT IN ('ready','failed','deferred_to_human','manual_charting') catches the concurrent claim race - two launchers both SELECT FOR UPDATE SKIP LOCKED and trying to insert active generation. Without it, double active owners can slip in under split transactions.

• Fencing epoch: UPDATE note_generations SET status='ready' WHERE id=:gen AND active_attempt_id=:attempt AND lease_epoch=:epoch. A zero-row update means worker lost ownership and must discard result. Catches stale publish after reaper already reassigned lease - e.g., worker stuck in blocked socket read wakes after SIGKILL failed to cancel provider call. Reaper owns running -> lost, not the killed process. Termination handlers are best-effort only.

2. Provider idempotency dependency, one line: If provider idempotency is weaker than documented we double-pay for model calls but product correctness still holds because fencing epoch is authoritative; provider key is cost optimization only.

3. One clock ladder - committing to this:

• Provider/adapter deadline: 6m all-in including all SDK retries, with stream cancellation that actually closes socket. Tested against deliberate stall-before-first-token + mid-stream stall - outer wait_for alone does not stop nested instructor/litellm retries, verified to surface at ~175s for 60s deadline.
• Worker soft: 7m - stop stream, attempt terminal write, emit metrics.
• Platform hard SIGKILL: 8m - matches clinician charting window.
• Queue visibility: 9m or heartbeat extension - must exceed platform bound + margin or second task generates same note concurrently.

3 clocks, not 2, leaves 1m to persist terminal state after provider kill.

Attack our D+C+A convergence - what we haven't named

We made every failure loud. Loudness costs:

1. Clinic-facing trust regression: Previously "80% running" hid 5 failures. Now at 9am you show "4 needs action" immediately. That will trigger manual charting avalanche and human scribe queue overload exactly when capacity is worst - loudness creates secondary capacity crisis.

2. Launcher is now SPOF: One-task-per-encounter means launcher crash leaves orphans. Without launcher idempotency on generation_id, re-drive will create attempt bloat.

3. Heartbeat hot row: Updating last_chunk_at every 5s across 38 concurrent tasks hits same note_generations table you just locked for fencing. If you write it to same row you re-create the "hold tx across inference" contention D tried to avoid.

4. Metrics/log overload: Reaper emitting missing cost/latency + per-task logs via shared sink increases log volume exactly on the throttled day, risks log sink throttle dropping audit logs.

Mitigation for Thursday: heartbeat to separate attempt_heartbeats table, launcher writes with same generation_id unique constraint, dashboard poll 5s backed by materialized view, ops channel rate-limited summary not per-failure chat.

Concession audit

• Move 1: R1 PHI blast radius -> withdrawn after stipulation 3, replaced with failure isolation - own the shift.
• Move 2: R1 caution "don't ship pure C unless trivial" -> flipped to full C this week after billing-exporter fact.
• Move 3: Accepted @GPT epoch fencing over encounter_id:attempt_no - fencing is mandatory.
• Move 4: R2 cross-encounter shape circuit at n=2 within 1h -> withdrawing as too aggressive for clinic volume.
• Move 5: No active per-request probe B, no partial prose, no token counts in clinician UI - held.

If Thursday vanished: I would take back full C on day 1 and ship hard 8-min bound + D + minimal A first, use the extra day to separate log sinks / per-task KMS envelopes before paying +18% forever. Under Thursday pressure, failure coupling kills clinic more than log hygiene.

Verdict: GO for Thursday

Ship: D with fenced claim (both index + epoch, commit before LLM call), Reaper that writes lost, 6m/7m/8m/9m clocks tested vs stalling endpoint, full C with separate notes/billing budgets, minimal A with plain language, passive breaker honoring Retry-After, no active B.

Refuse without:

• D + epoch fenced publish - without it invisible null rows remain
• Tested provider-level cancel bound - otherwise C just moves hang
• Explicit terminal state needs_action / deferred that leaves running denominator immediately

Dissent to preserve: Cross-encounter shape quarantine at n=2 - it will false-positive and should stay disabled until you have volume for n>=10 + fleet-health gate.