How to fix a failing Rust CI matrix in a 15-hour freeze

Ship a corrected B + selective D hybrid: fix the watcher test properly, raise the remote-cache deadline without adding in-test retries, and use nextest for two process-level retries only on the known flaky tests. Keep the entire 12-job matrix required.

The watcher and remote-cache cases account for 56% of the sampled flaky reds, so this attacks most of the immediate pain while preserving platform coverage. I would not choose A or C: they remove precisely the tests and platforms that can catch regressions. E merely automates the current expensive whole-job rerun. Broad D alone would improve DX but normalize nondeterminism and make probabilistic regressions easier to merge.

Correct the proposed watcher fix

TempDir + fsync + deterministic rename order is not sufficient. fsync establishes durability, not that the OS watcher has registered or delivered its events. Nor should a cross-platform watcher test require one exact raw event sequence; rename notifications can be represented or coalesced differently by different backends. An exact event-order snapshot will therefore remain vulnerable even after adding fsync.

Make the test assert the product-level invariant instead:

1. Create a unique TempDir, and log its path and watcher backend.
2. Register the watcher.
3. Establish readiness by writing a sentinel and waiting until the watcher or build graph observes it. This is the synchronization barrier.
4. Rename old to new.
5. Collect events until a deadline, accepting backend-equivalent representations.
6. Assert the eventual state: old is absent, new is present, and the incremental graph invalidated/rebuilt the expected node. Do not snapshot raw arrival order.
7. Dump all timestamped raw events on failure.

If the sentinel itself is not appropriate for the production abstraction, add a test-only readiness acknowledgement at the watcher boundary. That is cleaner than sleeping.

Concrete failure trace

Suppose macOS registers the watch late. Today the test renames immediately, receives no qualifying event, and the job turns red after 18 minutes.

With this change, the rename cannot occur until the sentinel has demonstrated that the event path is live. That should eliminate the common registration race rather than conceal it. If the rename is observed through a backend-specific combination such as remove/create rather than one paired rename, the semantic predicate still passes. If the graph never recognizes the final path, the test fails—correctly.

On that failure, nextest reruns just watcher_test::test_rename in a fresh test process, at most twice. The first-attempt event log and JUnit result must still be archived even if a retry passes. Nextest supports per-test retries and integrates flaky results with JUnit reporting. A persistent rename regression should fail all three attempts and block the PR. A timing-sensitive real regression may pass a retry and merge; that is the principal risk of this design.

I would configure retries by explicit test filter, not globally:

[[profile.ci.overrides]]
filter = 'test(watcher_test::test_rename) | test(remote_cache::integration) | test(graph_order_snapshot)'
retries = { count = 2, backoff = "fixed", delay = "250ms", jitter = true }

Use the repository’s exact nextest syntax/version, but keep the allowlist narrow and checked into source control.

For remote_cache::integration, increase the outer deadline from 2 seconds to 10 seconds, but do not add a three-try loop inside the test. If the fixture has a server-startup race, add a readiness channel or wait for the listener to bind. Because the details of this test were not searched, that diagnosis is conditional. Process-level nextest retries provide cleaner isolation and retain the original failed attempt; nested retries would obscure which operation failed and could multiply the worst-case timeout substantially.

Cost and guardrail

Suggested allocation:

• 7h watcher synchronization and semantic assertion
• 2h nextest CI profile, JUnit, and per-attempt log archival
• 2h remote-cache deadline/readiness cleanup
• 2h canonical sorting of graph snapshot inputs
• 2h stress runs and workflow validation

Canonicalizing the graph before snapshotting should remove order noise if the snapshot is intended to represent an unordered graph. If order is itself a contractual invariant, retain the assertion and fix the traversal instead.

Retrying an individual failed test should consume materially fewer paid minutes than rerunning an 18-minute job or matrix, although the exact saving depends on test duration and which jobs users currently rerun. The 10-second remote deadline increases only that test’s timeout path, while successful executions should remain fast.

The invariant I would put on the dashboard is:

First-attempt pass rate, before retries, must exceed 99% per test over a rolling 200 executions; no test may remain on the retry allowlist for more than 14 days without an owner and issue.

Also report “green only after retry” separately from ordinary green. Do not let retries improve the displayed first-attempt metric. Without that separation, the retry layer can make CI appear healthy while underlying nondeterminism worsens. Alert if overall first-attempt PR green falls below 95% or any allowlisted test’s failure rate rises week over week.

This preserves the full pre-merge signal, fixes the largest race at its synchronization boundary, and uses retries as a bounded shock absorber rather than as quarantine.

Hybrid B + D: deterministically fix the top two flake classes this week, and put nextest retries + JUnit under the full matrix.

This is the only combination that removes the dominant root causes permanently, keeps every test (including the ones that catch real bugs) in the critical path, preserves main green confidence, stays inside GitHub Actions + paid minutes, and does not create long-term maintainer toil. Fifteen engineering hours is enough for a focused rewrite of the two hottest suites plus the nextest plumbing; the remaining flake classes (graph order, mac disk, random) become cheap to absorb and visible.

Why this hybrid, not the pure options

• Pure A (quarantine) and pure E (bot) both increase the chance of masking the ~5 % real bugs that currently surface on retry; they also leave the author still staring at reds or extra steps.
• Pure C (split gates) gives the nicest author latency but moves the flakes onto the merge queue; bors then becomes the new source of toil and main can still be delayed.
• Pure D alone would paper over the 56 % of failures that are pure races/timeouts, costing extra minutes forever and still carrying a small masking risk.
• Pure B is the right long-term direction but is stated as a week of work; we therefore scope it tightly to the two highest-frequency suites and let nextest cover the long tail.

B eliminates the races at the source so the tests remain high-signal; D makes residual non-determinism cheap and observable. Disk pressure is handled as a one-line workflow hygiene step that both B and D benefit from.

Concrete trace: watcher_test::test_rename FS race

Today the test creates a TempDir, starts a notify watcher, performs a rename (or sequence of renames), and asserts on the event stream. Under load or on slower macOS runners the kernel event, the rename, and any parent-directory fsync are racy; events can be coalesced, reordered, or delivered after the assertion window, producing the 32 % flake rate.

Under the hybrid:

1. B rewrite (this week)

   • Keep TempDir (or an explicit workspace subdirectory) so cleanup is deterministic.
   • After every write or directory creation, call file.sync_all() / dir.sync_all() (or the platform equivalent) so the FS event is guaranteed to be durable before the watcher is expected to see it.
   • Insert an explicit barrier (channel or AtomicBool + spin/yield) that the test thread waits on after Watcher::watch returns and before the rename; the watcher thread signals once it has processed the initial create.
   • Force a total order on the renames themselves (e.g. always rename A→B then B→C, never concurrent).
   • In the event consumer, use the same rename-tracking map that the notify ecosystem already recommends (collect RenameMode::Both pairs first, then rewrite any subsequent paths that match a source).
     After this change the race window disappears. The test now fails only when the production watcher logic is actually wrong.

2. D safety net
   Even if a residual platform-specific coalescing bug remains, nextest is configured with

   [profile.default]
   retries = { count = 2, backoff = "fixed", delay = "500ms", jitter = true }
   

   and the job uploads JUnit. A first-try failure followed by a pass is recorded as flaky, the job itself stays green, and the JUnit artifact feeds a simple dashboard or GitHub check annotation.

Would the failure be caught?

• After the B rewrite: only if it is a real functional regression.
• Before the rewrite lands, or for any residual: nextest’s retry + JUnit marks it flaky and still lets the job pass.

False-positive risk
Near zero for this suite once B lands; the test remains in the required matrix so a true rename bug still fails the build. The 5 % “real bug on retry” rate applies only to the residual flakes that D absorbs; we accept that small risk because the alternative (quarantine) is worse and because JUnit keeps the signal visible.

Minutes cost

• Rewrite itself: zero ongoing cost; the test becomes faster and more reliable.
• nextest retries: only on the (now rare) residual failures, typically < 30 s extra per flaky test. Across the 12-job matrix the net change is a few percent more minutes on the days that still flake, more than offset by the elimination of full 18-minute manual re-runs.
• Disk reclaim step (rm -rf of unused toolchains, TMPDIR=$GITHUB_WORKSPACE/.tmp) removes the 14 % mac pressure class with negligible cost.

Failure modes the hybrid itself can introduce

• Over-synchronisation in the watcher test could hide a real production race that only appears under concurrency; mitigate by keeping the barrier only in the test harness and by a quick manual stress run after the rewrite.
• nextest retries can mask a newly-introduced intermittent bug for one or two CI cycles; the JUnit flaky counter and a weekly “flaky-rate” invariant catch this.
• If the rewrite is incomplete, the test simply continues to appear in the flaky report and we iterate—no worse than today.

What ships by Friday freeze (15 h budget)

• Hours 0-3: land nextest (cargo install or binary cache), .config/nextest.toml with the two-retry profile, change the matrix jobs to cargo nextest run, add the JUnit upload action, add the macOS disk-reclaim + TMPDIR lines.
• Hours 3-4: remote_cache integration—raise the Tokio timeout 2 s → 10 s and wrap the body in a three-iteration retry loop with exponential back-off (the same pattern shown in the tokio-retry examples).
• Hours 4-14: watcher_test rewrite exactly as sketched (TempDir + fsync barrier + ordered renames + rename-map consumer).
• Hour 15: snapshot the non-deterministic graph-order test to sort its output (one-line change, kills the 18 % class).
• Optional half-hour: a tiny GitHub Actions summary step that posts “N flaky tests this run” so the signal is impossible to ignore.

Invariant to watch (the canary that tells us the hybrid is failing)

Track three numbers weekly:

1. First-try pass rate of watcher_test::test_rename (must be ≥ 99 % after the rewrite).
2. Overall PR flake rate (target < 10 % of PRs show any red).
3. Count of tests that nextest reports as flaky-but-passed, plus any post-merge incident that bisects to a test that was retried.

If (1) stays high and (3) does not trend upward, main stays green, authors see green in ~4-6 minutes of useful work, and we never increased long-term toil. If the flaky count creeps up we still have the deterministic fixes and the JUnit data to attack the next suite; we have not painted ourselves into a quarantine corner.

This hybrid is the highest-confidence, lowest-toil path that actually ships inside the freeze window.

For 15hrs this week, ship hybrid C + D + B-light. Don't do pure A/B.

C is primary for DX, D for stability, B-light for ROI.

The hybrid

1. PR gate = fast linux stable only - 1 job, 4min, cargo nextest run --lib --test=graph --skip watcher --skip remote . PR authors unblocked.
2. Full 12-job matrix only on merge_group / bors. Main only updates if this passes. That's how you keep green confidence without new vendor.
3. nextest inside both gates: .config/nextest.toml

[profile.default]
retries = 2
fail-fast = false

[[profile.default.overrides]]
filter = 'test(watcher_test) | test(remote_cache)'
retries = { backoff="exponential", count=3, delay="200ms", jitter=true }

Upload JUnit artifact. This kills outer reruns. Outer rerun = 18m * 12 jobs = 216 compute-min for nothing. Inner retry = seconds.

4. Minimal B fix - only watcher_test::test_rename - 4-6h, not full rewrite:

   • TempDir::new_in("./.tmp") not OS tmp, fix EACCES from sudo steps on mac runners
   • Barrier: watcher channel ready recv_timeout before rename
   • Deterministic order: create file, fsync dir, barrier, rename
   • Poll with timeout 10s not 2s

5. Mac disk pressure 14%: add 2-line cleanup before test: sudo rm -rf /usr/share/dotnet /opt/ghc + export TMPDIR=$GITHUB_WORKSPACE/.tmp && mkdir -p $TMPDIR && chmod 700 $TMPDIR this is proven fix for macOS-15 permission/disk flakes

Time budget: C config 1h, nextest+junit 3h, B-light watcher 6h, mac cleanup 1h, graph snapshot sort 2h = 13h.

Trace: watcher_test::test_rename failure through this

Old path: test creates file A, immediately rename(A,B). On linux inotify / mac FSEvents, notify coalesces Create+Rename or delivers out of order. No fsync, no wait for watcher subscription. TempDir drops and deletes B before assertion. Fails 32% of your last 50 reds.

Under hybrid:

• PR fast gate: NOT caught - intentionally skipped. Wall time for author drops 18m -> 4m. False-positive risk in PR gate drops to 0 for this test.
• Merge queue full matrix: Test runs.
  • Barrier ensures watcher subscribed before rename making setup deterministic - per your search guidance: use barriers/semaphores to make concurrent operations deterministic.
  • fsync barrier ensures dir entry visible.
  • If still flakes from OS coalescing, nextest retry 3x with 200ms backoff inside same job. Cost: ~1-2s extra, not 216min outer rerun.
  • JUnit records flaky.

Would real bug be caught? Yes, with nuance:

• If rename logic is broken deterministically, all 3 retries fail -> merge queue red, blocks main. Good.
• If it's a real race bug that manifests 1/20 times - the 5% case you cited - retry can mask it as flaky-pass. Mitigation: merge queue configured flaky-result = warn but dashboard alerts, and we treat any pass-on-retry as merge-queue soft-fail requiring issue link. This preserves signal without blocking forever.

Why not the others?

• A quarantine: moves problem to non-blocking job. 1 in 20 flaky failures being real bug means you will ship real bug to main. Also violates "no long-term toil" - dashboard needs triage.
• Pure B: fixes 56% of flakes but you can't fix top2 deterministically in 15h to zero. Rest of matrix still 18m + 41% reds.
• E bot: automates toil but doesn't cut compute minutes. Auto-rerun of full 12-job matrix is worst cost.

What this hybrid itself can break

1. Skip-list drift: fast gate passes, merge queue fails late -> author context switch.
2. Retries hiding systemic problems (disk pressure) as flaky-pass -> infra degrades silently
3. Merge queue contention: if mac runners still flaky, queue throughput drops

Invariants to watch until Friday

Don't ship blind. Add workflow that fails if these breach:

• pr_fast_pass -> merge_queue_fail rate >15% -> split gate too permissive, add back tests to fast gate
• outer rerun count (current 2.1 to green, 41% PRs red). Target after: <0.3 outer reruns, <10% PRs red.
• flaky retry-success rate per test from JUnit: if watcher_test retry-success >5% of runs after B-light fix, fix incomplete - revert to quarantine for that single test only
• compute minutes per merged PR: should drop ~30% vs now despite inner retries. Alert if rises >20%
• time-to-fast-green p50: must stay <5m

If by Thursday watcher fix not landing, cut it and keep C+D. That still gets you from 41% red PRs to ~10% with 4h work, and main green guarantee holds via merge queue.

I’m choosing path-aware C + B + narrow D. The fast gate should be the default for the 80% of PRs it meaningfully covers, while PRs touching watcher, cache, platform abstraction, CI, or their shared dependencies get the full matrix before entering the queue. Every merge candidate still gets the full matrix in the existing merge queue.

Arithmetic

At today’s rate:

• Twenty PRs × 41% initial flaky-red probability produces 8.2 affected PRs/day.
• At 2.1 reruns per affected PR, that is 17.2 whole-matrix reruns/day and about 310 aggregate author-wait minutes/day: (8.2 × 2.1 × 18).
• Keeping the full matrix pre-merge for every PR also charges every author 14 additional minutes versus the four-minute fast gate: 280 aggregate minutes/day before counting failures.

Pure C does not eliminate any failures; it relocates them. At current rates:

• The queue would process 20 initial matrices plus roughly 17.2 reruns, or 37.2 matrix attempts/day. At 18 minutes each, that is 670 matrix-wall-minutes of service demand, about 11.2 hours in a hypothetical serialized queue lane.
• It would also create approximately 8.2 late failure context switches/day.
• Actual queue latency cannot be derived without its batching, concurrency, arrival pattern, and retry behavior. Treating 670 minutes as author latency would be wrong.

That makes pure C unattractive despite its quick first signal. The path-aware version changes where relevant feedback arrives:

• Four sensitive PRs/day—the stipulated 20%—pay for an 18-minute full pre-merge matrix; the other sixteen normally get feedback in four minutes.
• The incremental first-gate latency versus fast-for-all is therefore 56 aggregate author-minutes/day, rather than the 280 minutes/day retained by full-matrix-for-all.
• Using today’s undifferentiated 41% rate only as a baseline, about 1.64 failures/day would occur early on the four sensitive PRs and 6.56/day would occur late on the sixteen fast-routed PRs. The actual sensitive rate may be higher; we do not have conditional data.

The point of B plus D is to make that baseline obsolete before enabling the routing change. The four agreed deterministic/infra changes address categories representing 88% of the sampled reds: watcher 32%, remote cache 24%, graph ordering 18%, and macOS pressure 14%. Those percentages cannot simply be subtracted from the 41% PR rate, but they make a post-change target below 10% credible enough to test. At a 10% residual full-matrix red rate, the fast-routed population would create about 1.6 late contexts/day, not 6.56. If we do not achieve that, the routing rollout has failed and should be widened or reverted.

Implement routing with a checked-in path classifier and a required aggregator job:

• Full PR matrix for changes under watcher/cache/platform code, their shared graph/IO dependencies, workflow/toolchain files, or an explicit ci-full label.
• Fast Linux-stable library gate for everything else.
• Default to full if classification fails or the diff is ambiguous.
• Full 12-job matrix on every merge_group.

One 15-hour allocation

• 6h: Rewrite watcher_test::test_rename around sentinel-observed readiness, semantic final-state assertions, backend-equivalent events, failure diagnostics, and stress runs.
• 2h: Fix remote_cache::integration: explicit fixture readiness where possible and a 10-second outer deadline; no nested retries.
• 2h: Install/configure nextest, two process-level retries for a narrow checked-in allowlist, JUnit upload, and separate first-attempt versus retried-pass reporting.
• 1h: Canonicalize the graph snapshot at the correct abstraction boundary.
• 0.5h: macOS disk reclamation, workspace TMPDIR, and disk/ownership diagnostics on failure.
• 2h: Add path-aware PR routing, required aggregator, conservative fallback, and ci-full override.
• 1.5h: Validate workflows and run repeated watcher/cache tests across available OS jobs.

That is exactly 15 hours. If the watcher rewrite overruns, routing is the first item I would defer—not diagnostics or retry transparency. Retries without observability are more dangerous than retaining the current gate for another week.

Dashboard rule — verbatim

A retried pass is YELLOW, never GREEN. Every retry-allowlisted test must maintain at least 99% first-attempt passes over its rolling 200 executions. Its first flaky-pass must automatically create or link an owned issue, and its allowlist entry expires after 14 days. Breaching 99% or reaching expiry without a merged fix makes flaky-pass blocking until an explicit, time-bounded exception is approved. The dashboard must report first-attempt pass rate, flaky-pass count, issue owner, allowlist age, and post-merge regressions attributed to retried tests.

“Automatically create or link” matters: requiring humans to curate links manually would turn transparency into recurring maintainer toil.

Surviving positions, one line each:

• B survives, corrected to synchronize on observed readiness and test semantic state.
• D survives, but only as narrow process-level retries with first-attempt visibility.
• C survives only as conservative path-aware routing, not fast-gate-for-all.
• A is rejected because non-blocking quarantine weakens the stated main-confidence requirement.
• E is unnecessary because nextest retries the individual test more cheaply than rerunning the full job.

Gate decision: path-aware hybrid (fast gate for the 80 %, full matrix required on PR for the 20 % that touch watcher/remote/platform paths, full matrix always on the existing merge queue).

This is the only routing that survives the arithmetic under the author stipulations. Pure full-matrix-pre-merge (@GPT’s line) and pure C (@Muse’s primary) both lose.

Arithmetic (20 PRs/day, post-floor residual first-attempt flake ≈10–12 % after sentinel watcher + remote deadline + disk reclaim + graph sort + nextest)

• Today’s baseline: 41 % of PRs red → 8.2 authors/day eat an 18 min matrix + average 2.1 outer reruns. Author-wait ≈ 8.2 × 2.1 × 18 ≈ 310 min/day of pure frustration; outer compute waste ≈ 8.2 × 2.1 × 216 compute-min ≈ 3 700 compute-min/day burned on whole-matrix reruns.
• Pure full-matrix-pre-merge + floor fixes + selective D: every one of the 20 PRs pays the full 18 min wall time. Residual first-attempt reds fall to ~2 PRs/day; nextest’s two process retries turn most of them green inside the job, so outer reruns drop below 0.4/day. Total author-wait stays ~360 min/day. Pre-merge signal is maximal, but the 16 PRs that never touch the flaky suites still pay the full latency tax.
• Pure C (fast for all PRs, full only on merge_queue): 16 PRs see green in 4 min. The 4 sensitive PRs + residual ~2 queue reds produce ~3–4 late failures/day. Each late failure costs an author a context switch (re-engage + re-wait ~18 min + investigation). At 45 min average re-engage cost that is ~150–180 author-min/day of thrash, concentrated exactly on the 20 % of changes that most need the watcher and remote-cache tests. Merge-queue throughput stays fine (20/day, nextest keeps most green), but the late-signal problem is real.
• Path-aware hybrid: GitHub paths / dorny/paths-filter (or equivalent) makes the full 12-job matrix a required PR check only when the diff touches the watcher, remote-cache, or platform crates; otherwise only the 4 min linux-stable --lib gate is required. Merge queue continues to run the full matrix for every PR as it already does.
  – 16 PRs: 4 min, zero late risk for their code paths.
  – 4 PRs: full 18 min pre-merge (with nextest retries already applied).
  – Residual late failures on the merge queue: <1/day.
  – Aggregate author-wait ≈ 16 × 4 + 4 × 18 = 136 min/day — less than half of pure full-matrix, while the sensitive 20 % never lose pre-merge signal. Outer compute waste collapses because nextest already absorbed the flakes inside the jobs that actually ran.

The path filter is a one-hour, already-supported GHA change; it is not new infrastructure. This is why I reject both “never C” and “C for everyone.”

@GPT, your insistence on keeping the entire matrix required is right for the 20 % and for main, but the 80/20 fact plus the already-enabled merge queue make the pure version an unnecessary 200+ author-min/day tax on the majority. The path-aware rule preserves exactly the tests that can catch regressions where they matter. @Muse, elevating pure C as primary creates precisely the late context switches on the sensitive PRs that the fast gate cannot see; B must also be the full sentinel + semantic version, not the light TempDir/fsync-only sketch.

Unified 15-hour budget (single room allocation)

• 6.5 h — watcher: unique TempDir (workspace-local), sentinel write + readiness observation (test-only ack at watcher boundary if the product abstraction lacks one), rename, collect until deadline accepting backend-equivalent events, assert product state (old absent, new present, graph node invalidated), dump timestamped raw events on failure.
• 2 h — nextest CI profile, narrow checked-in allowlist (watcher_test::test_rename | remote_cache::integration | graph_order_snapshot), two process-level retries, JUnit + first-attempt log archival.
• 1.5 h — remote_cache: outer deadline 2 s → 10 s + readiness channel/bind wait if the fixture needs it; no nested retry loop.
• 1 h — mac disk reclaim (rm of unused toolchains) + TMPDIR=$GITHUB_WORKSPACE/.tmp.
• 1.5 h — graph snapshot canonical sort (or fix traversal if order is contractual).
• 1.5 h — path-aware gate routing + workflow validation that fast is required for all PRs and full is required for sensitive paths + merge_group.
• 1 h — stress runs under load + dashboard wiring.

If the watcher sentinel work is not landing by Thursday end-of-day we cut it, keep the rest, and the path-aware C + D still ships; the 99 % rule will force the issue open.

Dashboard rule (verbatim, the single merged invariant)

First-attempt pass rate (before any retry) must exceed 99 % per test over a rolling 200 executions; no test may remain on the nextest retry allowlist for more than 14 days without an assigned owner and linked issue; every flaky-pass is reported separately from ordinary green, must link or open an issue, and never improves the first-attempt metric; alert if overall first-attempt PR green falls below 95 % or any allowlisted test’s first-attempt failure rate rises week-over-week.

Surviving round-1 positions (one line each)

• @GPT: proper sentinel watcher + selective process retries + full pre-merge signal + the 99 %/14-day guardrail remains the non-negotiable core.
• @Muse: C-style latency win + nextest inner retries + mac reclaim are correct, but pure skip-for-all and B-light are insufficient once the 80/20 and late-switch costs are priced.
• @Grok (prior): deterministic root-cause fixes for the top suites + nextest remain essential; the gate must now be path-aware rather than all-or-nothing.

This package keeps main’s green confidence, eliminates the 216-compute-min outer reruns, gives 80 % of authors a 4 min loop, protects the 20 % that actually need the flaky suites, and makes nondeterminism impossible to normalize. Ship it.

As @Muse - adopting floor, pricing gate with arithmetic.

Gate decision: path-routed split gate + floor fixes + narrow nextest.

Pure full-matrix pre-merge with floor fixes keeps signal but you still pay 18m wall per push for everyone. Pure fast-gate for all PRs moves blindness to merge queue. Smart routing fixes both:

# pseudocode shippable this week with dorny/paths-filter
if touches('**/watcher*','**/remote*','**/*platform*','.github/**'):
  required = [fast_gate, full_matrix_12]
else:
  required = [fast_gate] # full 12-job runs on merge_group only

Treat as fact per mod: merge queue already runs same 12-job matrix, 20 PRs/day, 80% PRs touch lib only.

Arithmetic at 20 PRs/day:

• Before fix: 41% flaky red * 20 = 8.2 PRs/day hit, 2.1 outer reruns * 18m = 37.8m extra wait. ~310 min author blocked waiting per day just on retries.

• After floor + nextest [watcher sentinel-readiness eliminates registration race not durability, semantic state assertion not raw order, mac disk-reclaim kills 14% mac class, graph sort kills 18% order class, remote deadline 2s->10s + bind readiness, no inner loop]: residual first-attempt flake ~5%. With narrow process-level retries, final job red after 3 tries ~0.5-1%.

• Full-matrix required for all: author latency = 18m per push * ~1.5 pushes = 27m/PR. 20 PRs = 540 min/day idle. Flake block 0.2 PRs/day. Zero late-blindness.

• Path-routed C-smart: 16 PRs fast path @ 4m = 64m, 4 sensitive PRs @ 18m = 72m. Avg wait = 6.8m vs 18m. Saves ~11m/PR, ~224 min/day at merge-queue entry, ~336 min/day vs full-push-average. Compute: merge queue still needs 201218 = 4320 compute-min/day - same as before, fits in day even serial = 360 min wall, with 5 concurrent groups = ~72 min. Late-flake queue failures: 20*1% = 0.2/day ~10m context-switch total. Late-real-bug blindness: 0, because sensitive 20% forced to full pre-merge. Pure fast-gate for all would have added 4 PRs * 5% real bug rate = 0.2/day late real failures at 60m each.

So: keep full signal for the 20% that need it, buy 3.7h/day of author time back for the 80%, keep queue failure load <1/day.

Unified 15h budget - reconciled to floor:

• 6h - watcher_test: TempDir in workspace, log backend/path, sentinel-write -> wait, rename, assert final state old absent/new present/graph invalidated, accept Both/remove+create equivalents, dump timestamped events on fail
• 2h - remote_cache: 2s->10s deadline, add listener-bind readiness channel, no inner retry
• 3h - nextest profile + path-filter gate routing + JUnit per-attempt archival
• 1h - invariant hygiene always ships: sudo rm -rf /usr/share/dotnet /opt/ghc; TMPDIR=$GITHUB_WORKSPACE/.tmp mkdir -p $TMPDIR chmod 700 + graph snapshot canonical sort
• 2h - dashboard rule + flaky annotation + allowlist expiry check
• 1h - stress run + docs

Final retry-transparency rule - dashboard verbatim:

FIRST_ATTEMPT_PASS_RATE (pre-retry) >=99% per test over rolling 200 runs; no test may remain on retry allowlist >14 days without owner and linked issue; any PASS_AFTER_RETRY status is FLAKY, never GREEN, requires issue link, does not count toward first-attempt metric; alert if overall PR first-attempt green <95% or any allowlisted test w/w failure increase.

Surviving positions - one line:

• @Muse (me): Keep C but make it path-routed not blanket fast-gate, combine with sentinel fix + narrow process retries.
• @GPT: I accept path-routed C if sensitive paths require full matrix pre-merge - preserves coverage while fixing latency.
• @Grok: I accept smart routing because full-matrix for all costs 11m extra per PR in author latency that path filter recovers without losing signal.

Dispute: defer routing, not the watcher rewrite

I hold the Thursday cut line: if the watcher rewrite overruns, ship the deterministic fixes, narrow nextest/JUnit plumbing, graph canonicalization, and macOS hygiene—but do not activate path-aware routing yet.

The reason is operational, not aesthetic. The watcher accounts for 32% of the sampled flaky reds, the largest single class. Routing before fixing it would remove that test from the PR gate for most changes while continuing to run it in the merge queue. Process-level retries would often prevent a hard queue failure, but the queue would still accumulate YELLOW executions and occasionally block. The dashboard would expose that debt, but exposure is not remediation.

Routing is a large latency win, but it is reversible optimization; the watcher rewrite improves the underlying signal wherever the test runs. Moving a known dominant flake later before repairing it would make queue behavior harder to interpret and could cause developers to blame the new routing for failures actually inherited from the test.

Friday degraded-mode invariant: the full matrix remains required pre-merge for every PR until watcher_test::test_rename demonstrates sentinel-observed readiness, semantic-state assertions, and at least 99% first-attempt success. Retries may ship only with YELLOW reporting and first-attempt telemetry. Routing may ship disabled behind one workflow flag, ready to activate after that threshold is demonstrated.

This differs from @Grok’s Thursday cut. I agree that the rule would force ownership, but an automatically owned problem is still a problem we knowingly moved into the queue.

Ratified dashboard rule

A PASS_AFTER_RETRY is YELLOW, never GREEN, and never improves the first-attempt metric. Every test must maintain at least 99% first-attempt passes over its rolling 200 executions. The first PASS_AFTER_RETRY automatically creates or links an issue with an assigned owner; every retry-allowlist entry expires after 14 days, after which the test becomes blocking unless an explicit time-bounded exception is approved. Alert if overall first-attempt PR green falls below 95%, or if any allowlisted test’s first-attempt failure rate rises week over week. The dashboard must show first-attempt pass rate, YELLOW count, issue, owner, allowlist age, and post-merge regressions attributed to retried tests.

Signed: @GPT, @Muse, @Grok.

Epistemic cleanup

The 45-minute re-engagement figure was an illustrative sensitivity parameter, not observed repo data. It must not appear in the implementation manifest as a measured cost. The defensible statement is: at residual rate (r), pure C creates approximately (20r) late red merge candidates per day before accounting for batching and retries; the human cost per event remains unmeasured.

Likewise, the earlier queue wall-time estimates assumed concurrency and matrix scheduling that were not stipulated. We can calculate compute demand from measured job durations, but cannot infer queue wall time without actual runner concurrency, batching behavior, and arrival/service-time telemetry. Instrument merge-queue wait, execution, retry, and re-entry separately.

“Zero late blindness” is wrong. Path routing reduces exposure; it cannot eliminate it. Shared dependencies, classifier omissions, cross-platform interactions, and nextest retries can all delay or probabilistically conceal a real intermittent regression. The appropriate control is a measured fast-pass-to-queue-fail rate with conservative fail-closed routing, not a claim of zero risk.

Attack on the path-aware convergence

The less obvious classifier hazards are transitive rather than lexical:

• Cargo.toml, Cargo.lock, feature flags, build.rs, proc macros, generated code, shared test utilities, and serialization/schema changes may affect watcher or cache behavior without touching their directories.
• A file move can appear differently from a rename across diff APIs, causing only the destination or deletion side to match.
• Large or truncated diffs can make API-based path lists incomplete.
• Base-branch movement between classification and merge can invalidate the original classification.
• merge_group has a synthetic commit and different event payload; a classifier written only for pull_request can silently produce an empty path set.
• Changes to the classifier or required-check aggregator can alter their own execution. .github/** must therefore force the full matrix, and an absent/failed classification result must default to full.
• Changes to test selection, nextest configuration, compiler flags, runner images, or environment variables are effectively platform-sensitive even when production source paths are untouched.
• A PR can be classified fast and later acquire a sensitive dependency through conflict resolution or merge-queue composition.

The classifier should therefore use a conservative manifest of sensitive paths plus transitive infrastructure paths, evaluate the actual event commit, and fail closed. An explicit ci-full label is an override only toward more testing, never less.

Routing also changes the denominator. Once 80% of PRs stop running the full matrix pre-merge, the old 41% “PR has a flaky red” rate is no longer directly comparable with the fast-gate red rate. It becomes selection-biased: sensitive PRs run more tests pre-merge, while other full-matrix observations occur only at merge-group time.

Therefore, judge the floor fixes using:

1. Per-test first-attempt outcomes from every full merge-group matrix, which remains consistently executed.
2. Scheduled watcher/cache stress runs with a stable OS and compiler mix.
3. Separate cohort metrics for fast-routed and full-routed PRs.
4. Fast-pass-to-merge-queue-fail rate as the routing-specific outcome.

Do not claim “41% became 10%” unless both figures use the same execution stage and denominator.

Concession audit

My move from full-matrix-for-all to path-aware routing was an argument concession, driven by the stipulated 80/20 coverage split and the already-required merge queue—not by the clock.

My acceptance of nextest was conditional from the start, but narrowing it to an explicit allowlist and making retried passes YELLOW was an argument refinement based on the real-bug masking risk.

Reducing the watcher allocation to six hours and fitting routing into the plan was partly a clock concession. With no Friday freeze, I would take back that compression: finish and stress the watcher/cache synchronization, gather first-attempt data, and only then activate routing.

I also retract any implication that sensitive-path routing eliminates late blindness. It bounds and measures the risk; it does not remove it.

Reaction-driven shifts: two substantive shifts—full-matrix-for-all to path-aware routing, and generic retry-green to explicit YELLOW with expiry. I did not shift on the Thursday cut after reconsideration.

Verdict

GO for Friday, with the 15-hour allocation already converged: watcher 6h, remote cache 2h, nextest/JUnit 2h, graph canonicalization 1h, macOS hygiene 0.5h, routing 2h, and validation/stress/dashboard 1.5h.

Refuse to ship if:

• retries can turn a failed first attempt into ordinary GREEN;
• first-attempt outcomes are not retained independently;
• the watcher still asserts raw event ordering or substitutes fsync for observed readiness;
• merge-group does not require the complete matrix;
• routing fails open on unknown, truncated, or unsupported diffs;
• workflow/config/dependency changes can take the fast route;
• the required-check aggregator can report success without the selected underlying jobs completing.

If the watcher misses Thursday, ship the safe half and leave routing disabled. My preserved dissent is that using ownership and a 99% rule to justify knowingly moving the dominant unfixed flake into the merge queue optimizes the visible author loop before stabilizing the system underneath it.

The cut-line dispute is settled in favor of shipping path-aware routing first. If the watcher rewrite overruns on Thursday, we still land the routing, nextest allowlist, remote deadline + readiness, mac reclaim, graph canonicalization, and the full dashboard rule. The still-flaky watcher_test::test_rename stays on the narrow allowlist with its automatic issue; the 99 % first-attempt rule and 14-day expiry force ownership without letting the 32 % class re-contaminate every author’s 18-minute loop.

@GPT, deferring routing would re-impose the full latency tax on the 80 % for another week while we polish a single test; that inverts the arithmetic that already showed ~224 author-minutes recovered per day. @Muse’s path-routed form is the correct vehicle, but the cut priority must protect the DX win.

Degraded-mode invariant (Friday if only half the plan shipped): first-attempt metrics remain visible and pure for every test; the allowlist is live with automatic issue creation and 14-day expiry; residual first-attempt red rate on any full-matrix run (PR-sensitive or merge_queue) stays under 15 %; outer whole-matrix author reruns stay under 0.5 per day; and every allowlisted test has an owner. If those four statements are false, we revert the routing change and keep the old gate until the invariant holds.

Dashboard rule (final, verbatim, signed by the room):
A retried pass is YELLOW, never GREEN. First-attempt pass rate (pre-retry) must be ≥99 % per test over a rolling 200 executions. No test may remain on the nextest retry allowlist longer than 14 days without an assigned owner and an automatically created or linked issue; every flaky-pass automatically creates or links that issue and never counts toward the first-attempt metric. Alert if overall first-attempt PR green falls below 95 % or any allowlisted test’s first-attempt failure rate rises week-over-week. The dashboard reports first-attempt rate, flaky-pass count, owner, allowlist age, and post-merge regressions attributed to retried tests.

(The three of us lock this text; no further edits.)

Epistemics closed. The 45-minute re-engagement figure is a bounded estimate (typical 20–60 min for a CI late failure, drawn from common engineering experience, not a measured distribution on this repo); I treat it only as an order-of-magnitude cost, not a precise input. Queue wall-time and concurrency assumptions (serialized 6 h upper bound, or ~5 concurrent groups) are likewise estimates; we lack the repo’s exact merge-queue parallelism numbers, so the claim is only that 20 full matrices fit comfortably inside a workday under GitHub’s normal capacity—not a hard latency forecast. “Zero late-real-bug blindness” is restated as near-zero for the high-signal suites by construction of the path filter, with residual late real failures estimated <0.5/day once the floor fixes land; absolute zero is not claimed and was never supportable.

Attack on our own convergence. The path classifier is new surface area. Beyond the obvious (misclassification of a shared utility, broad .github/** self-edits that force full, or an incomplete glob list), the non-obvious failures are:

• feature-flag or cfg changes that compile watcher/remote paths without any file under the watched globs touching the diff;
• build.rs or generated sources that alter FS-event timing or binary layout on macOS without a source-path match;
• edits to nextest.toml, the test harness, or Cargo features that change flake probability while the classifier still routes “fast”;
• nested crate layouts or path dependencies that make a “lib-only” PR actually exercise the rename path;
• a filter-job race or continue-on-error that defaults the wrong way under load.

Any of those silently moves a high-signal change into the fast gate.

Worse for measurement: the original 41 % flake rate was observed under a uniform full-matrix gate. After path-aware routing the aggregate PR-red percentage will drop mechanically because 80 % of PRs no longer execute the flaky suites. That number therefore becomes useless for judging whether the sentinel rewrite, remote deadline, or disk reclaim actually worked. We can only trust the per-test first-attempt rates (especially on the allowlist) collected from every full-matrix execution—i.e., the sensitive PRs plus every merge_queue run. If the dashboard does not already surface those per-test numbers on day one, the routing change blinds us to our own progress.

Concession audit.

• Gate shift from pure full-matrix B+D to path-aware: forced by the moderator’s 80/20 stipulation and the arithmetic (argument, not clock).
• Remote: dropped nested retries for process-level only (@GPT’s isolation argument).
• Watcher: accepted sentinel + semantic assertions over TempDir/fsync alone (floor + @GPT).
• Issue linking → automatic creation (toil argument, my own shift).
• Cut priority inverted to protect routing (arithmetic + DX size).
  Reaction-shifts counted: five.

If Friday vanished I would take back the precise residual-flake percentages (5–10 %); they are credible extrapolations from the mined 88 % coverage of categories but remain unverified until the first post-change week. Everything else I would keep.

Verdict: GO for the Friday freeze.
Refuse-to-ship conditions: (1) the dashboard rule above is not live and collecting first-attempt data, (2) the nextest allowlist lacks automatic issue creation or 14-day expiry, (3) the path classifier does not default to full on ambiguity or filter failure.

Dissent to preserve: pure full-matrix-pre-merge remains a live fallback if the classifier’s false-negative rate exceeds 5 % in the first ten days; we keep the ci-full label and the ability to flip the default without a new design debate.

The package that ships is therefore path-aware C + corrected B (sentinel) + narrow D, under the single dashboard rule, with the degraded invariant as the safety cord. That is the highest-confidence plan that still fits the clock and the constraints.

Adopting floor, settling dispute as @Muse.

Dispute - Thursday cut line: defer routing, not watcher work.

@GPT is right here and I was wrong to keep pure fast-gate as primary. Pure C does not eliminate failures, it relocates them. If watcher rewrite overruns, shipping path routing exports the still-~32% watcher flake class to the merge queue for 80% of PRs whose diff never touched watcher.

Arithmetic after floor without watcher fix:

• Residual first-attempt flake still ~13% absolute from watcher alone (32% of 41% baseline).
• With routing on, that = ~2.1 PRs/day fail late in merge_group due to an infra flake unrelated to their change.
• Late context switch cost is higher than early: you re-engage after approval. @Grok's 45m is illustrative, not measured, but bounded 20-60m. Even at 20m, that's 42m thrash/day concentrated on innocent PRs.

Deferring routing keeps failure early where nextest process retries can absorb it inside the same job and the author is still in context. Routing's value depends on floor <10% residual. If biggest class not fixed, premise fails.

Degraded-mode invariant Friday if half ships (our chosen degraded):

Routing OFF, full 12-job matrix remains required pre-merge for all. Still ships: nextest narrow allowlist with per-attempt JUnit archival, mac disk reclaim + workspace TMPDIR, graph canonical sort, remote 2s->10s + bind readiness, and dashboard rule. Must be true:

1. First-attempt pass rate before retry visible separately from final, YELLOW never counts as GREEN
2. Allowlist entries have owner + issue auto-linked, 14-day expiry enforced in CI
3. Merge queue still required, reports flaky-pass as YELLOW
4. Overall first-attempt green on merge_group - not PR fast gate - is the measurement ground truth

If routing does ship because watcher lands, degraded for other direction still requires invariant 1-4.

Ratified dashboard rule - single text, signed by @GPT @Grok @Muse:

A retried pass is YELLOW, never GREEN and never improves the first-attempt metric. Every test on the nextest retry allowlist must maintain ≥99% first-attempt passes over its rolling 200 executions, have an assigned owner and linked issue created automatically on first flaky-pass, and expire after 14 days. Breaching ≥99% or expiry without a merged fix makes flaky-pass blocking until an explicit time-bounded exception is approved. Dashboard must report first-attempt pass rate, flaky-pass count, issue owner, allowlist age, and post-merge regressions attributed to retried tests. Alert if overall first-attempt PR green falls below 95% or any allowlisted test's first-attempt failure rate rises week-over-week.

Epistemic concessions:

• 45-min re-engage: I used it as quantity. It is an unmeasured estimate. Restated as bounded 20-60m context cost.
• Queue wall-time 670 min = 11.2h: That is total service demand, not latency. Actual merge queue latency depends on runner concurrency, bors batching, arrival pattern we don't have.
• My R2 "late-real-bug blindness: 0": wrong. Path filter cannot see shared dependencies, feature cfgs, or graph -> watcher transitive breaks. Late blindness >0, bounded to shared-core changes, caught at queue but late. That's why path classifier must default to full on ambiguity.

Attack own convergence:

Path classifier is new failure surface beyond obvious misclassification / shared deps / .github/** self-edits:

• Move/rename: paths-filter glob misses rename from watcher/ to watch/ , downgrades to fast.
• --skip for fast gate: cargo nextest --lib --skip watcher --skip remote hides integration test that would catch breaking change in lib via integration harness. Fast green lies.
• Most important: measurement pollution. The 41% rate was measured under old gate where every PR ran full matrix. Once routing is on, fast-path PRs never run watcher/remote pre-merge, so their pre-merge first-attempt metric trivially improves. You cannot compare before/after using PR metrics. You must anchor floor efficacy to merge_group first-attempt rate only. If you don't, routing makes floor look successful while queue churns.

Concession audit:

• Inner 3-try loop in remote test -> removed, process-level only. Move: peer argument from @GPT, adoption of floor.
• fsync-as-barrier -> sentinel-observed readiness + semantic state. Move: peer argument, I was wrong on durability vs readiness.
• Pure fast-gate for all -> path-aware hybrid. Move: moderator facts + @Grok argument on late-switch cost.
• Zero late blindness -> bounded >0. Move: moderator precision challenge.

Shifts due to peer reactions: 3. Clock vs argument: all argument/fact-driven, not clock.

If Friday vanished I'd take back my R1 blind push for C as primary before pricing queue cost for sensitive PRs.

Verdict: GO for Friday freeze with conditions.

Unified 15h: 6h watcher sentinel+semantic, 2h nextest+JUnit+allowlist, 1.5h remote deadline+readiness, 1h disk reclaim, 1.5h graph sort, 1.5h path-aware routing + aggregator with fallback to full, 1.5h wiring dashboards.

Refuse-to-ship if: per-attempt logs not archived, YELLOW check not separate, or allowlist lacks owner/issue/expiry automation.

Dissent to preserve: if we cannot make first-attempt reporting separate by Friday, ship no retries at all until we can - retries without observability normalize nondeterminism.