Flaky Tests vs. Probabilistic Variance: A Mathematical Framework

Flaky Tests vs. Probabilistic Variance: A Mathematical Framework

When a test fails intermittently, the instinctive engineering response is to re-run it. For deterministic software, this is usually the right instinct — a flaky test is almost always an infrastructure problem: a race condition, a timing dependency, shared state between test runs. For AI systems, that same instinct can mask genuine, meaningful variance in system behavior, and re-running until green trains a team to systematically ignore a real quality signal.

Two Different Phenomena, One Symptom

Both flaky tests and legitimate probabilistic variance produce the same surface-level symptom: a test that passes sometimes and fails other times on ostensibly the same input. But they come from fundamentally different sources, and conflating them has real costs in both directions.

Flakiness is a property of test infrastructure, not the system under test — a database that isn't properly reset between runs, a network call with an inconsistent mock, a timing assumption that doesn't hold under parallel execution. The underlying system, if you could isolate it perfectly, would behave consistently. The test harness introduces the inconsistency.

Probabilistic variance is a property of the system under test itself — a language model sampling from a distribution at a non-zero temperature, an agent making genuinely different but equally valid tool-call sequences to accomplish the same goal, a retrieval system returning a different but equally relevant set of documents on a repeat query. The system is behaving as designed. The variance is real, not an artifact of the test.

Why the Distinction Matters

Treat real probabilistic variance as flakiness, and you re-run failing tests until you get a passing result, then ship — even when the failing result was a genuine, reproducible-in-distribution quality problem that a differently-sampled run happened to avoid. This is how regressions slip past a test suite that technically ran and technically passed.

Treat real infrastructure flakiness as probabilistic variance, and you build elaborate statistical tolerance into your evaluation pipeline to accommodate noise that has nothing to do with the AI system's actual behavior — masking real infrastructure problems that would be better fixed at the source, and adding unnecessary complexity and compute cost to your evaluation process.

A Framework for Telling Them Apart

Step one: isolate the variance source. Run the identical test case multiple times against a fixed model checkpoint, fixed prompt, and fixed infrastructure configuration — controlling for every variable except the model's own sampling behavior. If variance persists under these controlled conditions, it's coming from the model's probabilistic output, not the test harness.

Step two: characterize the distribution, not just the pass rate. Rather than treating each run as a binary pass/fail, score each run using the LLM-as-judge methodology from our post on mitigating evaluator drift, and examine the resulting quality distribution. Infrastructure flakiness tends to produce a bimodal pattern — mostly high scores with occasional complete failures unrelated to output quality (timeouts, malformed responses, connection errors). Genuine probabilistic variance tends to produce a continuous distribution of quality scores clustered around a central tendency.

Step three: test infrastructure-isolation directly. If suspected flakiness might be infrastructure-driven, run the same test suite against a mocked, fully deterministic stand-in for the AI component. If failures persist against the deterministic mock, the flakiness is in the harness, full stop — the AI system's own variance can't be the cause, because it isn't in the loop.

Step four: apply variance-aware thresholds, not fixed pass/fail. Once genuine probabilistic variance is confirmed, the right response isn't to eliminate it (you generally can't, and often shouldn't, since some variance improves output quality) but to build statistical thresholds around it — flagging a test case as a real regression only when its quality distribution shifts beyond what's consistent with normal sampling variance, using the distributional comparison approach from our post on non-deterministic regression testing.

The Cultural Cost of Getting This Wrong

Beyond the technical mechanics, there's a team-level cost to conflating these two phenomena that's easy to underestimate. Engineers who repeatedly see tests fail and pass on re-run — without a clear framework for which category a given failure belongs to — rationally learn to distrust the entire test suite. That distrust generalizes: once a team stops trusting its tests, it stops looking closely at failures at all, which is precisely the condition under which real regressions ship silently.

Building and communicating a clear framework for distinguishing infrastructure flakiness from genuine probabilistic variance isn't just a technical exercise — it's what keeps a team's relationship with its own test suite intact as the system under test becomes more probabilistic.

Instrumenting for the Distinction, Not Just Diagnosing It After the Fact

The framework above describes how to diagnose a failure after it's already happened, but the more efficient long-term approach is instrumenting test infrastructure to make the distinction visible automatically, rather than requiring manual investigation every time a test fails intermittently. This means tagging every test failure with metadata at the point of failure — was this run against a fixed model checkpoint or a live, potentially-updated one; did the failure mode look like a malformed response or infrastructure error, versus a low-but-plausible quality score; was this test run in isolation or under concurrent load that might introduce genuine infrastructure contention.

Teams that build this instrumentation in from the start spend meaningfully less engineering time re-litigating "is this flaky or real" for every individual failure, because the test infrastructure itself surfaces enough context to make the classification close to automatic in the majority of cases, reserving manual investigation for the genuinely ambiguous minority rather than every single intermittent failure that occurs.

Frequently Asked Questions

How do I know if a failing AI test is flaky or a real quality problem? Run the identical test multiple times against a fixed model checkpoint and fixed infrastructure. If variance persists under those controlled conditions, it's coming from the model's own probabilistic behavior, not test infrastructure — and should be evaluated as a distribution, not re-run until it passes.

Should we eliminate randomness from our AI system to make testing easier? Not generally — reducing temperature or otherwise constraining sampling can measurably reduce output quality in many tasks. It's usually better to build test infrastructure that correctly handles genuine variance than to constrain production behavior for testing convenience.

What does a bimodal failure pattern indicate? A pattern of mostly-high scores with occasional sharp failures (timeouts, malformed output, connection errors) unrelated to actual response quality typically points to infrastructure flakiness rather than genuine model variance, which tends to produce a more continuous distribution of quality scores.

Is it ever okay to just re-run a failing AI test? Re-running to distinguish infrastructure flakiness from real variance is reasonable as a diagnostic step. Re-running specifically until a passing result appears, then treating that as validation, discards real signal and should be avoided once you've confirmed the variance is coming from the model itself.

How much variance is "normal" before something counts as a regression? This depends on the specific task and system, but the right approach is establishing a baseline quality distribution under known-good conditions, then flagging statistically significant deviations from that baseline — not applying a fixed, one-size-fits-all tolerance threshold across every test case.

---

Adept Mayar's evaluation pipeline distinguishes infrastructure flakiness from genuine probabilistic variance using distributional analysis, not re-run-until-green heuristics. Explore Adept Mayar or read about AI-Native QA & Testing. Request an assessment of your current test suite's signal quality.