A self-learning loop
for flaky tests
Duo distills our development docs into context, shifts flaky-test fixes left, and closes its own knowledge gaps — getting better every cycle.
Today, every flaky test starts from zero
End-to-end tests flake on timing, selectors, and shared state. We retry, quarantine, and eventually fix them — but the fix lives in one merge request and one engineer's head. Duo and our agents never carry that hard-won context into the next review.
Knowledge stays tribal
How to stabilize a page object or wait correctly lives with reviewers — not where an agent can reach it.
Fixes land too late
Flakiness usually surfaces after merge, in nightly E2E runs — far from the change that introduced it.
Context never accrues
Each fix is one-off. Nothing distills it back into guidance the platform can reuse.
What if the platform
taught itself?
Turn every fix into durable context — then feed it straight back into review.
Distill the docs into context Duo can use
A scheduled pipeline reads docs.gitlab.com/development, extracts our end-to-end testing conventions, and compiles them into artifacts the AI applies on every change.
Fix flakiness before it ever merges
Content generated by AI should be seen as a starting point and verified before use.
Mine the fixes, find the gaps, write the docs
A second scheduled job classifies merged flaky-test fixes by root cause and checks each recurring pattern against the testing docs — opening a merge request wherever guidance is missing.
Every fix makes the next review smarter
Self-reinforcing
Each merged fix becomes documented context, distilled into the next skill and review pass. Knowledge compounds instead of resetting.
Shift-left by default
Fixes move from nightly E2E failures into the merge request, where they are cheapest to make and easiest to learn from.
Runs itself
Two scheduled jobs do the distilling and the gap-finding. No manual curation backlog, no doc rot.
Close the loop
Pilot on the E2E browser suite, prove it closes once, then expand to every flaky-test bucket.