01
The plan under review
A 15-engineer-hour Friday sprint to restore confidence in a Rust build-graph tool's 12-job GitHub Actions matrix. Forty-one percent of PRs had at least one flaky red and authors waited 18 minutes per run. Five forks were on the table: quarantine flaky suites, deterministically fix the top two, move most testing to a fast PR gate and full merge queue, add process-level nextest retries, or automate triage and reruns. The request required one shippable choice, a concrete watcher-race trace, and a measurable failure invariant.
Scenario context20 PRs/day; 18-minute full matrix; 41% of PRs with a flaky red; watcher FS race 32%, Tokio timeout 24%, nondeterministic graph order 18%, macOS disk pressure 14%; ~1 in 20 flaky failures exposed a real bug on retry.Figures as stated in the reviewed plan — no inferred savings.
02
What changed before ship
Fast-gate-for-all AND full-matrix-for-all
→
Path-aware routing via crossed paths: fast gate default; full matrix pre-merge for watcher/cache/platform/CI and classifier-uncertain diffs; full matrix always on merge-group
Fsync barrier + deterministic rename order
→
Sentinel-observed watcher readiness + semantic assertions
Retry loops inside tests
→
Process-level nextest retries with attempt identity preserved
Retried green counted as success
→
The ratified rule, signed verbatim; allowlist expiry auto-files the owned issue
Aggregate PR-red rate as scorecard
→
Per-test first-attempt rate on merge-group + scheduled stress + fast-pass-to-queue-fail as the routing-specific metric
Routing forever
→
Reversion trigger: classifier false-negative rate >5% over ten days flips back to full matrix, no new debate needed
Five competing options
→
One 15-hour hybrid budget with a documented degraded mode for either side of the open decision
04
Decision
Ratified ruleUnanimous
A PASS_AFTER_RETRY is always YELLOW and never improves first-attempt metrics; every test must maintain >=99% first-attempt passes over rolling 200; the allowlist expires after 14 days with an owned issue.
The ratified rule was signed verbatim with "no further edits." The open cut-line is recorded as a decision boundary, not chosen — each side carries its own degraded-mode invariant.
05
The open decision
Open decision2-1
If the watcher rewrite overruns Thursday, does routing still ship?
Defer routing — root-cause first2 votes
Exporting the largest known flake class into the merge queue makes innocent PRs pay later (~2.1 innocent late queue failures/day at the stipulated volumes) and corrupts the latency win. Held by Sol and Muse — Muse switched sides with its own arithmetic.
Ship routing first1 vote
The ratified 99% rule forces the watcher race into owned remediation regardless; the reversion trigger flips the system back automatically if the classifier degrades. Held by Grok.
Decision owner · principal