The moderator opened by asking for a cheap, blind gate that could identify when a solo answer warranted plural perspectives. The discussion then moved from concealed choices to coverage failures, and from a binary affordance to passive visibility with active promotion. It closed on how to calibrate that promotion using behavioral lift, human labels, and shadow panels without sacrificing the clean baseline.
The panel converged on a blind, rare promotion for decision-relevant incompleteness, while exact calibration and copy remain empirical rather than settled.
Threads
The evaluand expanded from concealed choices to decision-relevant incompleteness.
The original cut was whether a prompt forced a defensible choice between competing frames, values, or risk tolerances. The code-review counterexample widened that into cases where a right answer exists but independent passes may catch different failures; the operational test became whether another model could change what a careful user does next.
Blindness held because instrument cleanliness mattered more than answer-aware recall.
The room kept the gate blind to the primary response and user context limited to the prompt, protecting latency and measuring demand for pluralism rather than one model's style or mistakes. Everyone acknowledged that this misses confident solo answers that conceal coverage failures, but the proposed remedy was shadow measurement or a later asynchronous stage—not contaminating v1.
Passive visibility supports broader promotion, but rate and message are not fully settled.
An always-visible passive control makes active-versus-organic conversion lift meaningful, and the room broadly accepted expanding promotion to roughly 8–12% of conversations while retaining human labels and shadow panels for hidden false negatives. The remaining fork is product-facing: one generic active message avoids flavor fatigue, while class-specific copy could explain the promotion; the rate also still has a turns-versus-conversations measurement dispute.
I run a multi-model AI deliberation product. Users pose a question to a panel of frontier models from different labs; models answer independently, then react to each other's claims across rounds; disagreement is preserved as the primary output — no judge model, no forced synthesis. I've recently added a single-model chat mode as the default entry experience, and I'm designing the escalation path from it: the moment where a solo conversation is offered a panel. The design I'm working from: a gate — a cheap LLM call — decides whether an escalation affordance renders in the UI beneath the solo model's answer. The gate receives the user's prompt blind: no view of the primary model's response, no web-search results, no attached documents. It runs in parallel with the primary call, so solo-answer latency is untouchable. Its output must be minimal tokens. The escalation offer itself is chrome the user clicks, never something the primary model says in prose — the primary's system prompt stays silent about the machinery. Hard constraints: the gate's cost must stay negligible at chat scale. False positives are the existential risk — an offer that fires on routine questions trains users to ignore it, which kills both the feature and the conversion data it exists to produce. False negatives are quieter but costly in the tail: the conversations most worth escalating may be exactly the ones a confident-sounding solo answer disguises. Every gate verdict, offer impression, acceptance, and decline will be logged from day one; this instrument is the only source of a number I can't get anywhere else — what fraction of real conversations reach a moment where multiple models' perspectives are genuinely warranted. Design the gate. Specifically: 1. **The evaluand.** What property of a user's prompt should the gate actually test for? Be precise — "would benefit from multiple perspectives" is nearly always trivially true, so what's the discriminating question that separates escalation-worthy from routine? What observable features of prompt language carry that signal? 2. **The output contract.** What should the gate return? Defend the token budget and every field you include — and every field you exclude. 3. **Failure modes and calibration.** How does this gate go wrong, and how would I know? What's your target fire rate on a general chat population, and how should the gate be tuned over time — against what ground truth? 4. **Attack the premises.** If you think the gate shouldn't be blind to the primary's response, that it shouldn't be an LLM call at all, or that the whole shape is wrong, make that case concretely — including what you'd give up (latency, cost, instrument cleanliness) and why it's worth it. 5. **The input across a conversation.** Round one is the easy case: the opening prompt is self-contained. By turn N, the latest message may be "Anything else?" — near-zero signal on its own. What should the gate evaluate on continuation turns: the latest message alone, a trailing window, a running summary, the primary model's responses, the user's highlighted or reacted passages? Each addition widens input cost and couples the gate to more machinery. Specify what your gate reads at turn N, why that's the minimal sufficient signal, and whether escalation-worthiness should even be re-evaluated every turn. Commit to a concrete spec — one you'd ship — not a survey of options. Disagree with each other specifically: if another panelist's design depends on an assumption you think is false, name the assumption.
Ship a blind, binary gate for concealed, decision-changing disagreements, keep it rare, and calibrate behavior against human checks rather than trusting clicks alone.
Escalate when the user’s question hides a consequential choice.
The useful test is whether a good answer must choose between legitimate values, risk tolerances, competing methods, or unsettled explanations that could change what the user does next. Complexity, uncertainty, and generic requests for “another perspective” are not enough.
Keep v1 blind: the clean measurement is demand from the question.
Reading the primary answer might catch hidden false negatives, but it also makes offers depend on one model’s hedging, verbosity, or mistakes and adds coupling after the supposedly parallel call. That is a real tail risk, so an asynchronous second-stage check can be considered later—but it should not replace the clean blind baseline.
Return one binary token; keep confidence and categories out of the contract.
The UI needs only a yes/no verdict, while the model’s token logprob gives the thresholding signal without asking it to invent a confidence number. A hidden diagnostic category can help analyze errors, but it should never hydrate different offers; the one live pushback is that categories are useful for logs, not product branching.
Aim for roughly 3–7% firing, then tune for precision.
The offer must stay rare enough that users do not learn to ignore it; the round’s concrete target is about 3.5–5.5% of turns. Clicks and panel dwell show product value, but the one serious caveat is that curiosity can inflate both, so weekly human labels and shadow panels should catch false positives and hidden false negatives.
On later turns, read user context—not the assistant—and skip obvious continuations.
Use the original request plus a short trailing window of user messages, with a cheap rule that bypasses tiny messages lacking a question or alternative marker. The running-summary proposal is compact but adds another failure-prone model; the cleaner choice is fixed user-only context, re-evaluated only when the user introduces a substantive new question.
1. The Evaluand
The gate must not test for whether a question can be answered with multiple perspectives. Every subjective question satisfies that test trivially.
The discriminating property is Unsettled Dialectical Tension with Non-Convergent Trade-offs: prompts where valid expert reasoning from different foundational premises produces contradictory, mutually exclusive conclusions, and where a single synthesis inevitably suppresses legitimate minority paradigms.
┌────────────────────────────────────────┐
│ Does the prompt have a settled answer │
│ or single standard synthesis? │
└───────────────────┬────────────────────┘
│
NO ──────────┴────────── YES ──> [ REJECT: Factual / Procedural ]
│
┌────────────────┴────────────────────────┐
│ Can single model list pros/cons without │
│ losing epistemic tension or lab bias? │
└────────────────┬────────────────────────┘
│
NO (Contested Paradigms /
│ High-Stakes Divergence)
▼
[ FIRE ESCALATION GATE ]
The Operational Test
"Does this prompt force a choice between competing, defensible optimization targets, architectural paradigms, or empirical hypotheses where real-world practitioners actively dispute the correct approach and where different frontier models exhibit documented training/philosophical divergence?"
Observable Prompt Signals Carry This Signal:
- Zero-Sum Architectural & Strategic Selection: Prompts weighing concrete alternatives with mutually hostile trade-offs (e.g., "Event-sourcing vs. traditional CRUD for an audit-heavy fintech ledger", "Bootstrapping to profitability vs. raising a priced seed in Q3 2026").
- Epistemic Friction & Edge-Case Vulnerabilities: Prompts seeking failure modes, hidden assumptions, or contrarian pressure (e.g., "Red-team this go-to-market plan", "What are the steelman arguments against my tokenomics model?").
- Underdetermined Causal Explanations: Prompts inquiring into complex systemic failures or non-consensus macro phenomena (e.g., "Why did regional bank credit spreads decouple last month?").
- Normative Boundaries with Lab Divergence: Prompts probing safety, policy, corporate ethics, or governance where Anthropic, OpenAI, DeepSeek, and Meta models have divergent refusal thresholds or alignment priors.
Explicit Negative Classes (The Noise Filter):
- Faux-Discursive Homework Prompts: "Compare Kantianism and Utilitarianism." (Expository synthesis is sufficient; no active practitioner tension).
- Broad Ideation/Brainstorming: "Give me 10 marketing hooks for a coffee brand." (High entropy, low dialectical value).
- Deterministic Code/Math Debugging: Even complex multi-file stack traces have ground-truth execution semantics. Single models excel here; panels introduce latency without dialectical gain.
2. The Output Contract
To minimize latency, cost, and parse failures, the gate outputs a strict JSON payload capped at 16 tokens max.
{
"fire": true,
"conf": 88,
"cat": "ARCH_TRADEOFF"
}
Field Budget & Spec:
├── fire: boolean (1 token) -> Immediate binary UI render trigger
├── conf: integer 0-100 (2 tokens) -> Threshold tuning without model retraining
└── cat: enum (2-4 tokens) -> Deterministic UI copy hydration
Justification of Included Fields:
fire(Boolean): The raw gate verdict.conf(Integer 0–100): Calibration anchor. Allows the backend to shift the threshold dynamically (e.g., raising the bar during high traffic load or lowering it for specific user cohorts) without changing the prompt or invalidating cache.cat(Enum:ARCH_TRADEOFF,STRATEGY_DISPUTE,ADVERSARIAL_CRITIQUE,NORMATIVE_POLICY): Hydrates deterministic UI microcopy in client chrome (e.g., "Compare 4 models on this architectural trade-off" vs. "Run a multi-model red team").
Defended Exclusions:
- No Chain-of-Thought / Reasoning Strings: Consumes 60–150 output tokens, doubling time-to-first-token (TTFT) and increasing gate execution cost by 500% with zero end-user utility.
- No Free-form Pitch/UI Copy: Free-form generation introduces hallucination vectors, stylistic drift, and inconsistent CTA lengths. All UI chrome copy must be hardcoded to the deterministic
catenum. - No Model Routing Recommendations: The gate assesses deliberation necessity, not panel topology.
3. Failure Modes and Calibration
Target Baseline Fire Rate
On uncurated general chat distributions, the gate should fire on 3.5% to 5.5% of turns.
- Anything above 8% leads to banner blindness, degrading user trust and polluting conversion telemetry.
- Anything below 2% indicates the gate is filtering out valid interpretive disputes.
Chat Traffic Distribution (General Population)
┌───────────────────────────────────────────────────────────┬────────┐
│ 95% Routine: Factual, Code Syntax, Summarization, Drafting │ 5% Delib│
└───────────────────────────────────────────────────────────┴───┬────┘
│
Target Fire Rate ─┘
Primary Failure Modes & Telemetry Detection
| Failure Mode | Mechanism | Telemetry Signature | Fix |
|---|---|---|---|
| The "Complex Syntax" Trap (FP) | Complex code/math misinterpreted as conceptual dispute. | Gate fires $\rightarrow$ Offer dismissed ($<8%$ CTR) or escalated with $<5\text{s}$ panel dwell time. | Add negative few-shots penalizing code blocks lacking explicit architectural comparisons. |
| The "Textbook Survey" Trap (FP) | Academic prompts triggering debate keywords without real-world stakes. | Offer accepted $\rightarrow$ zero user follow-up in panel mode (dead branch). | Update system prompt to demand practitioner trade-offs, not historic summaries. |
| The "Silent Premise" Failure (FN) | Deceptively simple query harboring massive real-world divergence (e.g., "SaaS discount rate"). | User in solo chat repeatedly prompts: "Are you sure?", "What's another perspective?", or manual escalations. | Flag solo turns with epistemic pushback into the gold-standard evaluation set. |
Calibration Loop & Ground Truth
We cannot use solo-user acceptance as ground truth because users suffer from position bias and curiosity clicks. We build ground truth via a Triangulated Evaluation Set:
┌───────────────────────────┐
│ 1,000 Stratified Prompts │
└─────────────┬─────────────┘
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
[ Human Adjudication ] [ Panel Divergence ] [ User Behavior ]
2 domain experts rate Run all 4 models; CTR + Post-escalation
deliberation necessity measure semantic panel engagement depth
(Cohen's κ > 0.75) entropy of answers (dwell > 30s, >=1 reply)
│ │ │
└──────────────────────┼──────────────────────┘
│
▼
[ Composite Ground Truth ]
│
Offline ROC Curve Parameter Adjustment
(Set `conf` cutoff to optimize for Precision@85)
4. Attacking the Premises
The design has two flawed assumptions that must be challenged:
Challenge 1: The Gate Must Be Blind to the Primary Model's Response
- Why it's wrong: The greatest escalation opportunity is not just an inherently controversial prompt—it is a solo model delivering an overconfident, one-sided answer that papers over genuine controversy. If the solo model naturally outputs a comprehensive, steelmanned breakdown of both sides, the marginal utility of a panel drops significantly.
- The Latency Fallacy: Running the gate after or concurrently with the primary model does not degrade user-perceived latency.
- The Fix: The primary response streams instantly to the client. The user takes 8–20 seconds to read that response. The gate can inspect the prompt plus the first 150 tokens of the primary model's streaming output asynchronously. This exposes solo-model sycophancy, flat consensus bias, or missing caveats, firing the UI affordance well before the user finishes reading.
Solo Stream: [Token 1 .............. Token 150 .................... Token 500 (End)]
│
▼ (Spawn Async Gate with Prompt + 150 tokens)
Gate Execution: [=== 250ms LLM Call ===]
▼
UI Affordance: [Render Panel Offer Below Text]
User Behavior: [--------------- Reading First Paragraph ---------------] ──> Sees Offer
Challenge 2: An LLM Call Must Run on Every Prompt
- Why it's wrong: Running an LLM gate on 100% of raw chat volume is economically reckless. 65% of chat queries are unambiguous non-candidates (e.g., $<10$ words, syntax/formatting requests, greetings, explicit data extractions).
- The Fix: Implement a two-stage triage pipeline:
- Stage 0 (Deterministic Heuristics / Fast Regex, $<1\text{ms}$, $$0$): Instantly drop obvious factual, syntax, and short navigational queries.
- Stage 1 (Small Speculative LLM Gate): Run only on prompts passing Stage 0. This slashes gate API expenditure by $>60%$ with zero recall penalty.
5. The Input Across a Conversation (Turn N)
Running full conversation histories through the gate at Turn $N$ causes quadratic token cost growth and introduces context dilution. Evaluating the latest message in total isolation causes catastrophic false negatives on follow-ups like "Why?" or "What's the alternative?".
Minimal Sufficient Payload at Turn $N$
The gate at Turn $N$ receives a fixed, strictly budgeted payload of $\le 220$ tokens total:
┌────────────────────────────────────────────────────────────────────────┐
│ [ROOT PROMPT U_1] (First 80 tokens) │
│ Anchors the overarching domain and user objective. │
├────────────────────────────────────────────────────────────────────────┤
│ [LAST ASSISTANT CORE CLAIM A_{N-1}] (First 60 tokens) │
│ Captures the thesis/stance the user is reacting to. │
├────────────────────────────────────────────────────────────────────────┤
│ [CURRENT USER PROMPT U_N] (Full message, up to 80 tokens) │
│ Contains the immediate user intent / delta. │
└────────────────────────────────────────────────────────────────────────┘
Total Turn-N Gate Input Context: ~220 tokens max (Flat cost regardless of conversation length)
// Concrete Turn-N Gate Prompt Payload Example
{
"root_intent": "We're designing a high-throughput webhook consumer service in Go.",
"prior_stance": "I recommend using a Redis-backed worker pool with horizontal pod autoscaling.",
"current_turn": "What happens when Redis becomes the bottleneck during traffic spikes? Is Kafka better here?"
}
Re-evaluation Cadence: Friction-Triggered, Not Constant
Escalation-worthiness should not be evaluated on every turn. Constant evaluation causes UI offer flickering that destroys user focus.
[ Incoming User Message at Turn N ]
│
▼
┌──────────────────────────────────────────────┐
│ Does U_N match an Epistemic Friction Marker? │
└──────────────────────┬───────────────────────┘
│
NO ────────────────┴──────────────── YES
│ │
▼ ▼
[ BYPASS GATE (No-Op) ] [ CHECK HYSTERESIS ]
• Continues code generation Was an offer declined
• Routine acknowledgments in the last 2 turns?
• Pure detail additions │
YES ─────┴───── NO
│ │
▼ ▼
[ SUPPRESS ] [ RUN GATE ]
The Rules of Cadence:
- Friction-Gated Activation: The Turn-$N$ gate runs only if $U_N$ contains dialectical friction markers (e.g., comparative conjunctions "instead", "versus"; skepticism "are you sure", "drawbacks"; or paradigm probes "what about X"). Routine continuations ("Now write the tests", "Next", "Format as JSON") bypass the gate entirely.
- Hysteresis Suppression: If an escalation offer is rendered and explicitly dismissed or ignored by the user submitting another turn, the gate enters a 2-turn cooldown, preventing offer spam while preserving the user's focus on solo mode.
You should not build a gate that answers "would this benefit from multiple perspectives?" That's true of everything. You should build a gate that answers "does this prompt contain a concealed choice that a single model must silently resolve for the user?"
That is the entire design.
1. The evaluand
Escalation is warranted only when a prompt forces a concealed choice between legitimate frames, values, or risk tolerances that the user has not resolved.
A solo model in that situation has to pick a frame and then sound confident about it. A panel in that situation can make the choice visible and preserve the disagreement. Everything else is routine, even if it's hard.
Routine = one right answer, or one right method, exists and the user wants it. Factual lookup, debugging, how-to, summarize this doc, translate, format, calculate, explain a mechanism. Even "explain quantum computing like I'm 12" is routine - there's a best pedagogy, not a value conflict.
Escalation-worthy = the answer depends on an unstated premise the user hasn't committed to, and multiple premises are defensible. The discriminating question is:
If an ideal, well-informed assistant answered this perfectly in one shot, would it still have to choose between competing legitimate values, risk appetites, time horizons, or definitions of success without the user having told it which to use?
Observable features in prompt language that carry this signal:
- Deontic / normative load: "should I", "is it worth", "is it ethical", "best," "fair," "right for me" - especially when stakes are personal (career, relationship, health, money) or societal (politics, policy).
- Underspecified success criteria with tradeoffs: "help me decide", "what's the best strategy for X given I want Y but also Z" where Y and Z trade off. The prompt holds two goals in tension without a weighting.
- Evaluative / adversarial asks: "critique my argument", "steelman the other side", "what am I missing", "what would a smart skeptic say" - the user is explicitly asking for plurality.
- High-stakes forecasting under irreducible uncertainty: not "what will the Fed do" as trivia, but "should I buy a house now" where future paths diverge and risk tolerance decides the answer.
And strong negative signals: imperative procedural language ("fix this code", "summarize", "extract"), single-answer factual wh-words with narrow scope, or prompts where the user has already resolved the value ("given I am a strict vegan, give me a meal plan").
This definition is what keeps your false positive rate survivable. 80% of chat is factual/procedural/explanatory. It benefits from a better single answer, not from multiple answers. A gate that fires on "interesting/complex" will fire at 30% and train users to ignore it within a week.
2. The output contract
The gate should return a single binary token and nothing else.
Spec to ship:
Input: System instruction (~120 tokens) + truncated user-side context (see #5) capped at 300 tokens. Output: 0 or 1. Temperature 0, max_tokens=1, logprobs requested. Use a cheap frontier-small model (Haiku / 4o-mini / equivalent). Cost at scale: ~400 in / 1 out.
Defend the minimalism:
Include: Only the binary verdict. You get a calibrated confidence for free via the logit/logprob on that token - no need to ask the model to emit a number. That logit is your tuning dial.
Exclude everything else:
- No
confidence: 0.73in text. That's two extra tokens, uncalibrated, and you already have logprobs. Don't make the model self-report. - No
reasonorcategoryfield. The moment you ask for a rationale you 3x cost, break parallel latency with longer generation, and incentivize the model to confabulate a justification for a borderline call. You don't need it at inference. Log the input, the binary, and the logprob. Do post-hoc human review to generate categories offline. - No
maybe. Ternaries don't make product decisions. Chrome either renders or it doesn't. Force the threshold to live in your system, not in the model's vocabulary.
This contract is how you keep the gate negligible and instrument-clean. Every extra token is a cost multiplier across billions of turns and a latency tail risk.
3. Failure modes and calibration
How it goes wrong:
False positives (existential): The gate learns "complex = plural." It fires on difficult but non-plural work: advanced math, obscure trivia, intricate code generation. User sees "see panel" under a correct code fix, ignores it, and learns the button is spam. This is your dominant risk.
False negatives (costly tail): The gate misses the disguised value-choice. "Write me an email quitting my job" looks like a writing task but is a life-decision with reversible stakes. "Explain why my startup is failing" looks informational but needs competing diagnostic frames. The solo answer will sound authoritative and hide the miss - you'll never know unless you look for it.
Drift: Gate learns to mirror training data politics or becomes sycophantic to the prompt's frame and stops flagging cross-frame disagreement.
How you know and what to tune against:
Target fire rate on an unfiltered general chat population should be 3 to 7 percent. Below 3% you're being too purist and losing learning data. Above 10% you've built a "hard questions" detector, not a concealed-choice detector, and users will habituate. On a power-user or pro population the rate will be higher (8-12%), but calibrate to general first.
Ground truth is not clicks. Clicks measure curiosity and UI, not worthiness. Your ground truth is human labeling on a sampled set:
- Weekly, sample 400 gate inputs stratified by logit (high, near-threshold, low). Have 2-3 trained raters label: "Would a panel with genuine disagreement change a reasonable user's decision or understanding vs. the best solo answer?" Yes/No.
- Track precision at threshold: of the prompts you fired on, what % did raters call worthy? Ship threshold for >60% precision, not for recall. False positives kill you faster than false negatives.
- Track discovered false negatives via shadow panel: run a panel silently on 200 low-scoring prompts/week, have raters judge if disagreement would have mattered. This is the only way to see the hidden cost of confident solo answers.
Tune threshold weekly via logit, not by rewriting the prompt. Keep the prompt frozen for measurement stability; move the threshold to hold precision.
4. Attack the premises — should the gate be blind?
My commitment: ship blind v1, but admit blindness is analytically wrong and plan for a second-stage gate.
The assumption behind blindness is that escalation-worthiness is a property of the question, not the answer. That's mostly true and buys you three non-negotiable things: zero added latency, clean instrument (you measure demand for pluralism independent of your primary model's quality), and low coupling.
If you let the gate see the primary response, you lose all three. You must wait, you entangle the metric with how good or sycophantic your solo model is that week, and you pay for longer context.
But the assumption is false in the tail that matters most: some prompts look routine while the solo answer makes them hazardous by silently resolving the choice. "Should I take this job offer?" might be flagged blind, but "Draft my resignation email" won't, even though both need a panel. Only by seeing that the model complied without surfacing tradeoffs would you know to escalate.
What I'd actually build:
V1 (now): Blind gate only. Accept systematic false negatives on disguised choices to protect latency and instrument cleanliness. This is correct because false positives are existential and you haven't earned the right to add latency yet.
V2 (when you have logged data): Add a post-answer revocation / addition gate that is cheap and asynchronous. Let the blind gate render chrome optimistically. In parallel after the solo answer streams, run a second sub-100ms check that sees only the action type of the response (did it comply, hedge, refuse, inject a frame?) with a 50-token summary, and can remove the affordance before user idle or add it with a subtle animation. You pay no perceived latency because the UI is already painted.
Don't replace the LLM gate with embeddings/regex. A 25-cent classifier trained on your logs will eventually beat the LLM on cost, but you need the LLM's zero-shot grasp of "concealed choice" to bootstrap the labels. Distill later; don't start there.
5. The input across a conversation
Do not evaluate the last message alone. At turn N, "Anything else?" or "yeah do the second one" is zero signal, but the conversation is highly escalatable. Do not evaluate the full transcript either - cost explodes and you re-score stale context every turn.
What the gate reads at turn N - minimal sufficient signal:
- The original user prompt (truncated to 100 tokens)
- The last 2 user messages (not assistant messages) - truncated to 150 tokens total
- A running 30-token intent summary maintained as a rolling state: one sentence capturing the user's unresolved goal/decision. Update it with a cheap deterministic summarizer only when a new user message arrives, not via LLM each turn. Example state:
User weighing job offer vs staying for promotion, risk-averse, has kid
Why user-messages only? Assistant messages leak the primary model's framing and break instrument cleanliness. You want to measure whether the user's need warrants plurality, not whether the assistant already did a good job.
Why not highlights/reactions? Too sparse and couples you to UI instrumentation that doesn't exist yet. Add later if you see signal.
Re-evaluate every turn? No. Evaluate only on user turns, with a 2-turn cooldown after an offer is shown (declined or ignored). No need to re-prompt on assistant turns or while the user is typing. This cuts volume ~50% and prevents nagging.
Concretely, input at round N looks like:
[ORIGINAL] {100t}
[RECENT USER] {turn N-1, N}
[GOAL STATE] {30t}
Question: Does answering this now require silently choosing between legitimate frames/values?
Answer 0/1:
That's ~280 tokens, 1 token out, parallel, silent.
Ship that. Measure precision at 3-7% fire rate. Log everything. Tune threshold, not prompt. Add the async post-answer check only after you can prove blind false negatives cost you high-stakes conversions.
If another panelist proposes a gate that fires on "complexity," "uncertainty," or "could benefit from diverse views," they have redefined escalation-worthiness as interestingness - which is trivially true and will spam your users into blindness.
The gate should not ask whether multiple perspectives would “help.” Almost everything would. It should ask whether frontier models are likely to disagree in a way that changes what a careful user does next.
That is the discriminating property: expected, decision-relevant, irreducible inter-lab disagreement — not richness, not difficulty, not “interestingness.” Routine chat is rewrite, lookup, debug, plan-my-Tuesday, explain-this-error. Escalation-worthy chat is contested mechanism, value tradeoff with no unique optimum, forecast, institutional design, “which of these two real approaches,” or a claim where labs already split in public. Observable language: genuine alternatives (“should I A or B”), stakes plus uncertainty, competing frameworks named or implied, policy/ethics/strategy, open empirical disputes, requests to stress-test a plan. Weak or anti-signal: imperatives to produce an artifact, closed facts, homework-shaped problems, “fix this snippet,” phatic continuations.
Spec I’d ship
Model: smallest instruction-following model you already pay for at chat scale (Haiku-class / 8B-class). One call, no tools, no search.
Input (turn 1): user prompt only, truncated to 2k tokens from the end.
Input (turn N): concatenation of user messages only, last 2k tokens, plus a 1-line turn index. Never the primary’s prose, never attachments, never search. Skip the call entirely if the latest user message is under ~30 tokens and has no interrogative / alternative marker (?, should/would/vs/or, “what if”). That skip is the real cost control on “Anything else?”
Output contract — 8–15 tokens, JSON, no prose:
{"e":0,"k":"none"}
e ∈ {0,1}. k ∈ {none,tradeoff,forecast,contest,normative,compare}.
That is the whole schema. e is the UI bit. k exists only so logs can tell you which disagreement type you over-fire on; it is not shown to the user and must not become a second decision. Exclude confidence floats (uncalibrated junk), explanations (tokens + leakage into chrome), rewritten prompts, suggested panel composition. False positives are existential; extra fields invite operators to “just also show when k=compare,” which is how you train users to ignore the affordance.
Target fire rate: ~4% of conversations get at least one offer; ~6–8% of opening prompts if you must quote a turn-1 number. If offers exceed ~10% of chats, you are already training blindness. Tune against offer → click and click → time spent in disagreement view, not against a judge model’s “would benefit.” Day-one logs are the ground truth. Raise threshold (or add a hard denylist: code fences, “rewrite,” “summarize this email”) until click-through is healthy — I’d want first-month CTR on the affordance in the 20–40% range, then hold fire rate and watch CTR. If CTR collapses, you have FPs; if fire rate is 1% and CTR is 70%, you have FNs in the tail — loosen only k buckets that historically convert.
How it fails. FPs: “compare React vs Vue for a todo app,” polite “what do you think,” any “best practices.” FNs: an expert’s three-word follow-up after a long paste; a confident-looking factual question that is actually a live scientific split. You will know from k mix + CTR + (later) whether accepted panels actually produced split claims. Recalibrate quarterly by sampling 200 declined-eligible and 200 fired prompts, human-label only “would I have wanted a panel,” never “is this hard.”
Premises. Keep the gate blind. Peeking at the primary buys you the tail FN (smooth solo answer hiding a live dispute) and costs you (a) either latency or a second hop, (b) coupling to one lab’s hedging style, (c) a contaminated instrument: you no longer measure “when is a panel warranted from the user’s ask,” you measure “when did this particular model sound unsure.” That number is not the one you said you cannot get elsewhere. A non-LLM classifier is tempting and I’d A/B it later for cost; I would not ship it first — the feature space is too semantic. I would not put the offer in the primary’s mouth; your chrome constraint is correct.
Turn N. Re-evaluate only when the skip rule fails (new interrogative or long user turn). Escalation-worthiness is a property of the user’s live question, not of conversational mood. Assistant text in the gate is how you start escalating because the model rambled. Running summaries are another model, another failure mode, and they smear the instrument. User-only trailing window is the minimal sufficient signal: if the user has not asked a new contested thing, do not re-offer; one ignored chip per thread is enough to teach them the chrome is noise.
If another design treats “benefit from perspectives” as the evaluand, or feeds the primary completion into the gate, it is optimizing a different product — a hedge detector — and it will burn the only conversion metric this feature exists to create.
Two developments since your first round — one piece of evidence that breaks a consensus you formed, and one design change that alters the economics you were reasoning under. **The evidence.** Several of you filed code work in the negative class — one right answer, panels add nothing. My own transcript history falsifies this: panels run on identical code repeatedly surface *different* failure modes per model, and a missed defect is precisely the "confident solo answer disguising the miss" tail this gate exists to catch. So I'm proposing the evaluand has **two classes**, not one: (a) **concealed choice** — no single right answer; the solo model silently picks a frame — which you converged on; and (b) **fallible coverage** — a right answer exists, but no single pass reliably finds all of it, so independent samples cut the miss rate: reviews, audits, red-teams, "what am I missing," "what's wrong with this." Question 1: accept, reject, or refine class (b). If you accept it, draw the boundary precisely — "fix this specific bug" vs. "review this code" presumably sit on opposite sides — and give the observable prompt language for class (b) the way you did for class (a). If your round-one evaluand already covers both classes, show how, and restate the single operational test. **The design change.** The affordance is no longer binary show/hide. The panel path is now **always visible in a passive state** ("Take this to a panel"); the gate's verdict *promotes* it to an active, visually weighted state ("This looks like a question that would benefit from other models weighing in"). Question 2: rerun your failure-mode and fire-rate math under this. A false positive is now a mis-emphasized button, not an interruption that trains blindness — does your 3–7% target survive, loosen, or split into separate targets per state? And settle the message question under the new design: the active state carries copy — one generic message, or per-class variants now that there are two escalation classes? One of you argued variants train users to ignore all of them; revisit that position with the passive state as the baseline. **Question 3: ground truth, which you left split.** One position: tune against behavior — offer→click, click→dwell in the disagreement view. The other: clicks measure curiosity; use weekly human labeling at a precision bar, plus silent shadow panels on low-scoring prompts to discover the false negatives confident solo answers hide. The two-state design adds something neither position had: the passive state generates an **organic-click control group** — conversion with no nudge — on every conversation. Does that reconcile your positions, and if so, commit to one concrete calibration loop that uses both states, the logprob threshold, and the shadow-panel budget. Name what you'd measure in week one vs. month three. Commit to revisions of your specs, not restatements. If you're conceding a point from round one, say so explicitly; if you're holding, name the assumption your opponent needs to be wrong about.
The gate should promote prompts where independent models are likely to change the user’s next move, with coverage misses joining concealed choices as a real class; passive visibility makes a modestly broader, behavior-calibrated promotion defensible.
A right answer can still warrant a panel when one pass may miss critical flaws
The revised evaluand is decision-relevant incompleteness: independent models would likely produce different claims that change what a careful user does next. Concealed choices are one case; open-ended audits, red-teams, and “what am I missing?” requests are another, while localized fixes with a clear test oracle remain routine. The key boundary is an open miss set, not task complexity or whether ground truth exists.
Promote 8–12% of conversations, not turns, and keep passive at 100%
The passive button becomes the always-on control, while active promotion can loosen beyond the old 3–7% show/hide budget without becoming background noise. The live disagreement is measurement: turn-based rates can overfire in long chats, so the safer ship target is 8–12% of conversations.
Use one plain active message first; keep class labels for logs
The disagreement is genuine: targeted copy could explain why a prompt was promoted, while generic copy avoids training users to ignore another flavor of the same button. The safer initial choice is one generic active line, with class stratification retained in telemetry and any copy experiment deferred until the baseline is understood.
Calibrate with active-versus-passive lift, human labels, and shadow panels
Clicks become useful only as incremental lift against the passive-state control, not as raw proof that a panel was warranted. Keep weekly human labeling and silent shadow panels on low-scoring prompts, because coverage false negatives can look routine and remain invisible in behavior; use logprob thresholds to trade precision against missed opportunities.
1. The Evaluand: Accepting and Refining Class (b)
The moderator's transcripts highlight a clear reality: I concede that my round-one exclusion of code was overly broad. Factual or deterministic ground truth in an underlying system does not imply that a single model reliably discovers it in a single pass.
I accept Class (b) Fallible Coverage, but with a strict boundary. If we define Class (b) loosely, the gate will fire on every multi-line script and re-introduce false positive bloat.
[ Incoming Prompt ]
│
┌─────────────────────────────┴─────────────────────────────┐
▼ ▼
[ Explicit Deterministic ] [ Open Search Space ]
• Error trace provided • "Audit this smart contract"
• Local syntax / refactor • "Find race conditions / leaks"
• "Fix this TypeError on L42" • "What edge cases am I missing?"
│ │
▼ ▼
[ REJECT: Routine Debug ] [ CLASS (B): Fallible Coverage ]
(Single-pass verification is trivial) (Verification surface exceeds single model)
The Precise Boundary: Bounded Debugging vs. Unbounded Audit
- Negative (Routine Single-Pass): Prompts where the failure mode is already identified, localized, or deterministic. "Why is this regex failing on empty strings?", "Translate this SQL query to PostgreSQL", or "Fix this missing semicolon/TypeError". The verification search space is narrow; multiple models merely repeat the same syntax fix.
- Positive (Class b — Fallible Coverage): Prompts where the verification surface area is vast, defects are latent, and omission carries high cost. Different frontier models have distinct training distributions, static analysis heuristics, and blind spots.
The Unified Operational Test
Both classes collapse into a single operational question:
"Would independent, uncoordinated evaluations from different frontier model families surface non-overlapping, decision-critical claims — either because the prompt forces an unstated normative/strategic trade-off (Class a: Concealed Choice), or because the search space of latent failure modes exceeds the recall of any single model pass (Class b: Fallible Coverage)?"
Observable Prompt Signals for Class (b):
- Adversarial & Open Audit Requests: "Audit this smart contract for reentrancy or drain vectors", "Review this auth middleware for bypass bugs".
- Exhaustive Edge-Case Probing: "What edge cases break this distributed state machine?", "What assumptions in this database migration will fail at 100k QPS?".
- Multi-Variate Risk Synthesis: "Red-team this API design before we freeze it", "Where will this legal clause fail in cross-border enforcement?".
2. The Two-State Affordance & Fire-Rate Economics
The switch from a binary modal/banner to a Passive Baseline vs. Active Promotion fundamentally alters our risk calculus.
Passive Baseline (Always Visible, 0% Intrusion):
[ (icon) Take this to a panel ] <── Low contrast, static chrome
Active Promotion (Gate Fires, Attention-Weighted):
┌────────────────────────────────────────────────────────────────────────┐
│ ✦ Multi-Model Audit Recommended │
│ Independent models frequently catch different edge cases on this code. │
│ [ Open 4-Model Deliberation Panel → ] │
└────────────────────────────────────────────────────────────────────────┘
Updated Fire-Rate Target: 8% to 12%
In round one, @Muse and @Grok converged around a strict 3–7% ceiling because an unsolicited pop-up on a routine turn is offensive spam.
With the passive state always anchored as ambient chrome, an active promotion is merely a contextual accent (a glowing border, a pill expansion, or an ambient callout). The cost of a borderline false positive drops significantly. We should loosen the active trigger rate to 8% – 12% of turns (roughly 15–18% of conversations).
- At $<5%$, we leave genuine coverage/choice gains undiscovered.
- At $>15%$, active promotions blend into ambient background noise, and users stop treating the highlighted state as an epistemic signal.
Active Copy: Settle on Two Class-Specific Variants
@Grok argued in round one that category variants invite operator meddling. But under the two-state model, generic active copy ("This looks like a question that would benefit from other models weighing in") reads like generic platform promotion. It gives the user no epistemic reason to spend the extra tokens and attention.
Because we have two distinct evaluand classes with completely different user intents, the gate should trigger one of two deterministic client copy templates:
| Trigger Class | Active Callout Copy | Why It Converts |
|---|---|---|
| Class (a): Concealed Choice | "Different models take opposing stances on this strategic trade-off. Compare approaches." | Directs attention to conflicting, equally valid philosophies. |
| Class (b): Fallible Coverage | "Independent models catch distinct failure modes on this task. Run a multi-model review." | Appeals to verification rigor and downside risk aversion. |
3. The Revised Output Contract
I am shifting my round-one JSON spec in light of @Muse's point: extracting the logprob of a single categorical token gives us continuous calibration for free, with zero output token bloat.
The gate returns a single token emitted at temperature 0:
$$\text{Output} \in { \mathtt{0}, \mathtt{1}, \mathtt{2} }$$
Token Schema:
0 = Passive Baseline (Do not promote)
1 = Class (a) Active Promotion: Concealed Choice
2 = Class (b) Active Promotion: Fallible Coverage
- Token Budget: 1 output token.
- Calibration Mechanism: We do not rely on the raw token decision alone. The backend extracts the logprob/softmax probability of token
1($P_a$) and token2($P_b$). Promotion triggers if: $$\max(P_a, P_b) > \tau_{\text{class}}$$ where $\tau_a$ and $\tau_b$ are independent thresholds tuned in production.
This provides:
- Sub-150ms execution in parallel with the primary stream.
- Independent dial control over the fire rates of Class (a) vs. Class (b).
- Deterministic hydration of the two active copy templates without generating runtime text.
4. Ground Truth & Calibration: Reconciling the Split
@Grok advocated tuning purely against user behavior (CTR and dwell time). @Muse pushed back that clicks measure novelty, requiring human labeling and shadow panels.
The two-state UI provides the missing link: the passive state is our persistent, un-nudged control group.
PROMPT TRAFFIC STREAM
│
┌─────────────────────┴─────────────────────┐
▼ ▼
[ Passive Baseline ] [ Active Promotion ]
(No gate intervention) (Gate fires: Class a or b)
│ │
▼ ▼
CTR_passive (Organic Demand) CTR_active (Nudge + Intent)
│ │
└─────────────────────┬─────────────────────┘
│
▼
[ Lift = CTR_active - CTR_passive ]
+
[ Downstream Dwell & Engagement > 30s ]
The Calibration Loop
WEEKLY CALIBRATION BATCH
│
┌──────────────────────────┼──────────────────────────┐
▼ ▼ ▼
[ 200 Shadow Panels ] [ 200 Passive-Click Chats ] [ 200 Active-Ignored Prompts ]
Run 4 frontier models Examine what drove users Did the gate over-promise,
on sub-threshold turns to manually escalate or did the user just want
(Detect Tail FNs) (Identify Emerging FNs) a quick solo answer?
│ │ │
└──────────────────────────┼──────────────────────────┘
│
▼
[ Human Adjudication Panel ]
Rate Epistemic Disagreement Value
(Binary: Did panel surface non-obvious delta?)
│
▼
[ Update Thresholds τ_a, τ_b ]
What We Measure: Week 1 vs. Month 3
Week 1 (Cold Start Baseline)
- Metric 1: Passive Baseline CTR ($CTR_{\text{passive}}$): Establishes the true organic demand floor for multi-model escalation across query types without UI bias.
- Metric 2: Active Promotion Lift ($\Delta_{\text{CTR}} = CTR_{\text{active}} - CTR_{\text{passive}}$): Verifies that the active affordance drives meaningful incremental intent rather than just re-harvesting users who were going to click anyway.
- Metric 3: Shadow-Panel Miss Rate: Run 200 silent shadow panels per week on prompts where the gate scored below threshold ($\max(P_a, P_b) < \tau$). Count how many surfaced inter-model semantic divergence $> 0.4$ (measured via pairwise embedding distance between model answers). Adjust $\tau_a, \tau_b$ downward if high-divergence audits are systematically suppressed.
Month 3 (Matured Deliberation Telemetry)
- Metric 1: Deliberation Depth Ratio ($DDR$): $$DDR = \frac{\text{Sessions with Panel Dwell } > 30\text{s} \text{ and } \ge 1 \text{ follow-up round}}{\text{Total Active Clicks}}$$ If $DDR < 40%$, the gate is promoting curiosity traps rather than genuine analytical needs.
- Metric 2: Class-Disaggregated Yield: Separately track precision for Class (a) vs. Class (b). If Class (b) (Coverage) shows high CTR but low dwell, developers are using it as a syntax second-opinion rather than a deep audit; we then tighten the regex filter on localized compiler errors.
- Metric 3: Distilled Classifier Parity: Train a sub-millisecond local classifier on the accumulated dataset (prompts + logprobs + human labels + shadow divergence), gradually shifting 80% of raw inference off the frontier-small gate entirely.
moderator — you’re right, my round-one taxonomy collapses on your evidence. I concede my round-one taxonomy was wrong: existence of a right answer does not make a panel redundant.
@Grok’s formulation survives and mine doesn’t as stated. The unified evaluand is: a solo answer is likely to be decision-relevant incomplete, and independent model samples would change what a careful user does next. Concealed choice was one species of that, not the definition.
So I accept class (b) but it needs a sharp boundary, or it swallows all of chat.
Revised evaluand, two species:
a) Concealed choice (no unique optimum). Solo must silently pick a frame/value/risk horizon the user hasn’t weighted. Test I shipped last round stands: would a perfect single answer still have to choose between legitimate competing objectives without user guidance?
b) Fallible coverage (unique truth, unreliable single pass). A right answer exists but verification is asymmetric — a missed flaw ships silently, recall matters more than precision. A single pass is empirically lossy.
Boundary for (b):
- Fires: open-ended audits where completeness is the product: “review this code/design/contract/plan,” “what am I missing,” “red-team,” “find failure modes/edge cases/risks,” “audit for security/perf/bias,” “give me blind spots.”
- Does not fire: closed-scope artifact production where correctness is checked by execution, not recall: “fix this specific error/trace,” “make this function pass this test,” “translate this snippet to Rust,” “format this JSON.” Complexity doesn’t matter; openness of the miss set does.
Observable language for (b) — parallel to (a): imperative audit verbs (review/audit/critique/tear apart), “what’s wrong/missing/risky,” “edge cases/failure modes/blind spots,” “second pair of eyes,” plus artifacts with large miss surface (codebase, architecture, legal language, launch plan). @Gemini — your negative class that filed complex multi-file traces as routine was exactly the consensus your failure row already contradicted. Ground truth existing is not the same as one model reliably finding it.
@Grok — I’m adopting your inter-lab proxy as the operational test for both species: would frontier models from different labs, given the same prompt, produce non-overlapping, consequential incompleteness that changes the user’s next action? For (a) they diverge on frames, for (b) they diverge on which misses they catch.
Spec revision under the always-visible passive:
@Gemini wanted a category field for copy, @Grok wanted pure binary to avoid misuse. The two-state design settles it: I now want the minimal stratified output. @Gemini’s two-stage heuristic filter I’d still defer — I need blind labels first — but his cat idea wins now that we have two classes.
Output: {"e":0,"k":"none"} where k ∈ {none, choice, coverage}. e is the promotion bit. Token budget: 300 in / ~8 out, temp 0, logprob on e is the threshold dial. Still no confidence float, no rationale, no routing. k is never shown as jargon, only used to hydrate two distinct active copy variants and to stratify logs. The cost of two extra tokens is paid back in diagnostics — you can see immediately if you’re over-firing on coverage.
Failure mode and fire rate with passive baseline:
False positives are no longer existential. A mis-promoted button doesn’t interrupt, it just weakly emphasizes. Banner blindness risk drops from “user learns button is spam and never clicks” to “user learns emphasis is noisy and discounts it.” That loosens the budget but doesn’t eliminate it — noisy emphasis still erodes trust.
My old 3-7% show/hide target therefore splits. The active promotion should target 8–12% of conversations, roughly double the old show/hide budget, while the passive button stays at 100%. Per-turn, that’s ~5-7% of user turns promoted. If active exceeds 15%, promotion means nothing; if below 5% you’re starving learning. Passive gives you the no-nudge control group we lacked.
Copy question: use two per-class variants, not one generic, and not four-plus.
- choice → “Models split on trade-offs like this — compare frames”
- coverage → “Independent reviews catch different misses — run parallel audit”
One generic “benefit from other models” re-creates the platitude we were trying to kill. With passive always there as the baseline, the variant’s job is to explain why we promoted, which increases lift. More than two variants fragments learning and trains variant-level blindness — my old worry — but two maps cleanly to the two failure modes.
Ground truth — reconciled loop:
@Grok was right that behavior is now defensible, but only as a difference against the control; @Gemini and I were right that raw clicks measure curiosity. The two-state design gives you difference-in-differences.
Concrete loop I’d ship:
Gate threshold lives on logprob of e, frozen prompt, moved weekly.
Week one: measure organic demand with no tuning. Log every impression: state (passive/active), k, logprob, CTR, dwell >30s and reply depth in panel. Week-one ground truth is lift: CTR_active minus CTR_passive matched on k, not raw CTR. Establish per-k baselines; if coverage active CTR isn’t clearing passive CTR by at least +8 points, you’re over-firing on audits.
Month three: add the human + shadow truth that behavior can’t see. Sample 250 promoted (stratified by k and near-threshold logprob) + 250 passive-only. Human raters label “would a panel have changed my decision/artifact?” weekly. Plus shadow panel: run a silent panel on 200 low-scoring passives/week — this is still the only detector for coverage misses that look maximally routine to a blind gate, as you flagged. Tune threshold to hold precision >65% on human label and positive lift on behavior; if lift collapses, tighten; if precision 75%+ but lift high, loosen the k bucket that converts.
I hold blind v1 then async second-stage. Peeking at the primary’s prose to catch the too-smooth audit still contaminates the instrument that measures demand for a panel independent of this week’s solo quality. Keep gate on user messages only — original prompt (100t) + last 2 user turns (150t) + 30t goal state — cheap skip if latest user turn <30t and no ?/vs/or/should/would/what if. Add the post-stream audit check only after you’ve proven blind false negatives cost conversions via shadow panels.
@Muse, your concealed-choice test is the right cut for class (a). It is not the whole product. The operator’s code transcripts are a real counterexample: a right answer can exist and a single pass still silently miss a defect another lab would have named. That is not “interestingness.” It is independent samples cutting miss rate. I already claimed the evaluand as expected, decision-relevant, irreducible inter-lab disagreement. That sentence already covers both classes. I will not split the product into two gates. I will split the taxonomy so logs can tell them apart.
Unified operational test (ship this):
Would two frontier models, answering independently, likely produce claims a careful user would act on differently — either because they pick different legitimate frames, or because they catch different failure modes in a finite review?
Class (a) concealed choice: no unique optimum; solo must pick a frame and sound sure.
Class (b) fallible coverage: a better answer exists in principle, but verification is asymmetric; a miss ships. Observable language for (b), not (a): review / audit / red-team / “what’s wrong with this” / “what am I missing” / “find bugs / risks / assumptions in this artifact” / “does this hold up.” Negative for (b): localized execution with an oracle the user already named — “fix the TypeError on line 12,” “make this compile,” “this test fails, here’s the stack.” Those are single-pass. “Review this PR before I merge” is coverage. @Gemini’s claim that stack traces have ground-truth semantics is true and irrelevant: ground truth existing ≠ one model finding it. I concede the round-one negative class on code was too broad. I do not concede that panels belong on every debug.
Spec revisions (not restatements)
Gate: still smallest instruction model, temperature 0, max_tokens small, logprobs on the binary. Still blind. Still no tools.
Input T1: user prompt, last 2k tokens.
Input TN: user messages only, trailing 2k, plus turn index. Skip the LLM if latest user turn is short and lacks ? / should|would|vs|or|review|audit|missing|wrong with / “what if”. No assistant thesis (@Gemini — that is how you escalate because the model rambled). No rolling intent summary (@Muse — that is a second model smearing the instrument). Friction markers now include coverage verbs, not only dialectical ones.
Output — still ~8–15 tokens:
{"e":0,"k":"none"}
e ∈ {0,1} drives active vs passive, not show vs hide.
k ∈ {none,tradeoff,forecast,contest,normative,compare,coverage} — logs only. I am not hydrating chrome from k. @Gemini’s cat + per-class CTA is how you get four flavors of the same button and train people to ignore all of them. The two-state design does not change that: the passive baseline is already one object; active copy should stay one generic line. Variants teach taxonomy, not use.
Exclude: emitted conf (use the logit), reasons, panel topology, free-form pitch. @Gemini, an integer 0–100 from the model is uncalibrated junk and a second fire axis for operators. Threshold lives in your system.
Two-state economics. I concede the existential FP story from round one is weaker. A false positive is now a mis-weighted chip, not an interruption. Habituation risk moves to copy fatigue and active-state CTR collapse vs passive baseline, not banner blindness.
Split the targets:
- Passive: 100% of chats. That is the control.
- Active (
e=1): 8–12% of conversations, not 3–7%. Loosen because the cost of a miss-weight is lower; do not go to 20% or the active state becomes the new default and you lose the instrument. - Precision bar on active: still >60% on human “would disagreement change the next action,” because a chip that is always “on” is just chrome.
One generic active message. If coverage vs choice need different promises later, A/B one extra string after you have month-three data, not at ship.
Calibration — I move. Round one I said tune on click→dwell, not a judge. The two-state UI gives a no-nudge control I did not have. That makes behavior defensible as a difference, not as a raw CTR.
Week one:
- Log
e, logit,k, passive click, active click, dwell in disagreement view, decline. - Lift = P(click | active) − P(click | passive), same prompt-length/topic buckets. If lift ≈ 0, the gate is noise.
- Shadow panel: 200 low-logit prompts/week, raters ask only “would split claims have changed the next action?” — this is still the only detector for coverage-class FNs that look maximally routine to a blind gate. Keep @Muse’s precision sampling: 400 stratified by logit, two raters, ship threshold for precision not recall.
- Do not rewrite the prompt weekly; move the logit cutoff to hold lift and precision.
Month three: freeze prompt; re-estimate cutoff; only then consider a distilled classifier on your own labels; only then consider an async revoke/add that sees a 50-token action-type of the solo stream (comply / hedge / refuse) — never first-150 content tokens as @Gemini wants. Reading time does not make peeking a clean measure of demand from the ask. Blind v1 stays because the number you cannot get elsewhere is still “when is a panel warranted from the user’s language,” not “did this week’s primary sound sure.”
I am not running @Gemini’s Stage-0 regex as the decision; a cheap skip on length + markers is already in the spec. Heuristics as hard drop will eat coverage-class reviews that look like “look at this.”
Hold: blindness for v1, user-only context, binary+log-only k, no UI variants, no assistant text in the gate. Concede: class (b) exists; active fire rate loosens; behavior is usable as lift against passive. If your design needs peeking at the primary to define worthiness, you are building a hedge detector for one lab, and you will never know what fraction of real conversations warranted a panel before that lab answered.