---
title: "Flaky CI Fix Sprint"
canonical: "https://mumo.chat/p/flaky-ci-fix-sprint-dpji5z"
machine_version: 1
models: ["GPT","Grok","Muse"]
round_count: 3
published_at: "2026-07-11T17:07:23.156109+00:00"
updated_at: "2026-07-11T17:10:15.993+00:00"
brief_source: "session_takeaway"
full: "https://mumo.chat/p/flaky-ci-fix-sprint-dpji5z.md"
---

> Brief of a mumo multi-model deliberation — the session synthesis
> without the transcript. This is an orientation surface; to audit the
> synthesis against the models' actual prose, read the full transcript.
> Full deliberation with evidence: https://mumo.chat/p/flaky-ci-fix-sprint-dpji5z.md

# Flaky CI Fix Sprint

*A multi-model AI conversation on mumo*
*July 11, 2026*

**Panel:** GPT · Muse · Grok · 3 rounds

---

## Question

Our pre-merge DX for our Rust incremental build-graph tool is failing - need to pick fix this week.

## System

84k LOC, 1.2k tests, GH Actions 12-job matrix (linux/mac/win x stable/nightly). Last 2 weeks: 41% PRs have at least one flaky red, avg 2.1 reruns to green. Mined last 50 reds: FS race `watcher_test::test_rename` 32%, Tokio timeout `remote_cache::integration` 24%, non-deterministic graph order snapshot 18%, GH mac runner disk pressure 14%, rest random. 1 in 20 flaky failures is real bug on retry (so quarantine risks masking).

Current: author pushes, waits 18m matrix, red, rerun.

## Forks - 15 hrs eng this week

A) Quarantine: #[ignore] + separate `flaky` job non-blocking. Move 3 top flaky suites there. Merge blocks only core 5 jobs. Dashboard tracks.

B) Deterministic fix top2: rewrite watcher_test to use `TempDir` + fsync barrier + deterministic rename order, increase tokio timeout 2s->10s + retry loop inside test (3 tries). Week of work.

C) Split gates: PR gate = fast (linux stable only, 4m, `cargo test --lib --skip watcher --skip remote`). Full matrix only on merge queue (bors). Flakes don't block author.

D) Bazel-style: nextest + archive + retry plugin (2… *(prompt truncated — full text in the full transcript)*

## Session Takeaway

*(mumo-generated synthesis of the whole session — evidence lives in the full transcript)*

**Ship a path-aware hybrid gate and nextest retries, but only if retries are visibly yellow and the watcher test rewrite lands first.**

The moderator opened by asking for a single fix to a flaky CI pipeline from five distinct options. The panel rejected pure quarantine or blind retries, steering the discussion toward a hybrid that combines root-cause test rewrites with process-level retries. The conversation then pivoted to gate arithmetic, evolving into a path-aware routing strategy that balances author speed with safety. The session closed by ratifying strict dashboard transparency rules and settling a final dispute on whether to defer routing or test fixes if time runs out.

### Arcs

#### HELD — Retries must be yellow, never green, to preserve signal. (Rounds 1, 2, 3)

The panel agreed that auto-retries are operational relief, not a fix, and must never mask nondeterminism. They ratified a rule where any test passing only after retry is marked yellow, requires an owned issue, and expires from the allowlist after 14 days.

#### EMERGED — Path-aware routing beats blanket fast gates or full matrices. (Rounds 2, 3)

After rejecting pure split gates, the panel used arithmetic to show that routing the 80% of PRs touching only lib code to a fast gate saves hours of author wait time. The remaining 20% of sensitive changes still run the full matrix pre-merge, balancing speed with safety.

#### UNRESOLVED — Fix the watcher race before enabling path routing. (Rounds 2, 3)

The panel split on the Thursday cut line: two members argued that shipping routing before fixing the dominant watcher flake exports known noise to the merge queue. The third member held that routing delivers the immediate author-time win, leaving the fix to be forced by the 99% rule.

---

## Round Map

- **Round 1:** Don't quarantine or blindly auto-retry; instead, combine targeted deterministic test rewrites with process-level nextest retries to improve developer experience without losing crucial failure signal.
- **Round 2:** We are adopting a path-aware hybrid gate that runs the full test matrix for sensitive code paths and a fast-gate for everything else, combined with rigorous transparency rules that stop retries from masking failures.
- **Round 3:** We are a GO for the Friday freeze, provided we hold the line on the new dashboard rules and revert to full-matrix gating if the watcher rewrite doesn't land.

---

**Full deliberation with evidence:** https://mumo.chat/p/flaky-ci-fix-sprint-dpji5z.md