Aviation has never been sentimental about opinions. A design survives because test data supports it, and the day the data disagrees, the design changes. Software testing could borrow far more of that culture, because its arguments are still settled by habit in plenty of teams. Six beliefs in particular keep circulating despite published evidence to the contrary. Each appears below with the correction attached, and the field that studies them — running an automated suite the way any production service is run, with developers as its users and verdicts as its product — has collected under the name TestOps.
Six TestOps Myths, Put to the Evidence
Myth 1: More tests automatically mean more safety
The instinct is understandable: if some scrutiny is good, more must be better. Delivery research keeps finding the opposite correlation. What separates high-performing teams is not suite volume but confidence in verdicts — roughly ten checks that everybody believes outperform hundreds that everybody re-runs. A bloated suite also taxes every stage it touches: slower feedback, higher compute bills, and alerts so frequent that attention thins out. Watch how suites grow, too — every incident adds three tests, and no quiet year ever removes one. Volume without credibility is dead weight with a dashboard attached.
Myth 2: Flaky tests are a fact of life, so just re-run them
Arithmetic dismantles this one. Give each test a mere one-in-a-thousand chance of a false failure and line up a thousand of them: about sixty-three per cent of runs will come back red for no reason. Add a single automatic retry and the false-alarm rate collapses to roughly a tenth of a per cent — while conveniently concealing the defects that genuinely come and go. The retry-until-green habit is corrosive enough that Bazel discourages the flaky attribute in its own build system: marking a test flaky makes reruns official and investigation optional. Note, too, that flakiness has a workable definition: different outcomes on the identical commit. That turns a vibe into a measurement.
Myth 3: TestOps is just QA rebranded
The label is new and trendy, which invites suspicion, but the comparison fails on scope. It is not a department, not a purchasable product, and not the discipline formalised in the ISO/IEC/IEEE 29119 test-management standard. It does not set release policy either — deciding whether a failing check blocks shipping belongs to release operations. What it owns is narrower: stage budgets, flake detection, environments and data, sharding, test selection and results analytics, all in service of one question — does this verdict deserve trust? A new label on an old department changes nothing; a new operating model around an old department changes everything. The name is the smaller of the two claims.
Myth 4: UI-heavy suites are the most thorough
Testing everything through the interface feels like end-to-end rigour, and produces the shape practitioners call the ice-cream cone: a fat layer of slow, expensive, brittle browser checks balancing on thin unit coverage. The classic counter-shapes — pyramid, honeycomb, trophy — are frequently argued online as if one were correct. The evidence-backed reading is duller and better: pick the distribution by the cost and determinism of each layer in your product, not by which diagram is fashionable this quarter. Interface checks are the slowest to run and the least steady in their verdicts, which is precisely backwards for a layer you want gating every change.
Myth 5: Coverage percentages and test counts prove health
Google’s engineers published numbers in 2016 that should end this argument. Around one and a half per cent of their test runs were flaky; roughly sixteen per cent of tests displayed flaky behaviour; and eighty-four per cent of pass-to-fail transitions turned out to involve a flake rather than a real regression. Translation: a very large, well-funded suite still produced results in which most sudden failures were noise. Health is measured by different instruments — the share of red results that are false, and how long developers wait for an answer. Neither appears on the spec sheet of any testing product we know of, which tells you where the industry’s attention has been.
Myth 6: AI self-healing ends test maintenance
Vendor demos are impressive; independent evidence is thin. The strongest published result for machine learning in this field concerns selection, not healing: Meta’s models decided which tests to run, skipped more than two of every three eligible ones, and still let fewer than one regression in a thousand slip past — after being trained on data from which flaky results had been stripped first. Self-healing locators have no comparable independent track record. The defensible posture mirrors what our industry already teaches through AI for aircraft engine health monitoring and computer vision in aircraft inspection: automate the watch, log every intervention, and keep humans responsible for removal decisions and gate policy.
Where the Evidence Leaves TestOps
Stack the corrections together and a consistent picture appears. In compressed form:
- Suite size is not safety; confidence in each verdict is.
- Retries manufacture green screens and bury real intermittent bugs.
- The discipline is neither a QA rename nor a release gate.
- Test-shape diagrams are heuristics, not doctrine.
- Coverage tells you what is executed, never what is believed.
- Automate the watching; keep the judging human.
Teams that want the full research context — definitions, metric design, maturity stages, the lot — can read the Ops Atlas chapter on TestOps in one sitting.
A simple readiness test comes before any investment of effort. When the quick stage drifts past the ten-minute mark, when failed jobs get rerun on reflex, or when the main branch averages a noise-red every week, the suite needs operating discipline, and the ideas above repay themselves fast. If none of those apply, spend the energy on writing missing tests instead — that is the honest bottleneck. Operations-minded readers will recognise the mindset from how AIOps sharpens everyday IT operations: instrument first, judge second, and let evidence retire the argument. Aviation learned that lesson over a century of hard cases; software is welcome to copy the homework.
What actually changes day to day once the six beliefs are retired? Standups get shorter, because a red result carries information instead of starting an argument. Onboarding gets easier, because new engineers inherit a suite whose rules are written down rather than folklore. And releases stop feeling like dice rolls, because the people shipping have a number for how often their alarm system lies. None of that requires heroics; all of it requires deciding that the verdicts themselves deserve maintenance.