Six Signal-to-Noise Metrics for a Mobile Test Suite

Most mobile QA dashboards report volume: tests run, pass rate, coverage percentage. None of those numbers tell you whether the suite is doing its job, which is separating real defects from everything else that can turn a build red. A team can run ten thousand tests a week at a 98% pass rate and still be shipping bugs, because pass rate says nothing about what happened to the 2% — whether it was triaged in five minutes or five days, whether it was a real defect or a stale selector, whether anyone believed the result at all.

Signal-to-noise metrics measure the thing pass rate misses: how much of what the suite reports is worth an engineer's attention. Here are six worth tracking, in roughly the order they are easiest to start measuring.

Retriage rate. Of the failures a suite produced this week, what share were closed as "not a real defect" — a stale selector, a timing issue, an environment blip, a change nobody updated the baseline for? Track it as a percentage of total failures. A suite in the 60–80% range is spending most of its failure budget on noise; a well-validated suite should be well under 20%, because most of that noise was filtered before it ever reached an engineer.

Escape rate. Of the defects that shipped to production and were reported by users or support, how many should the test suite have caught, based on the coverage it claims to have? This is the metric that tells you whether high coverage numbers are real. A suite can report 90% coverage and still have a high escape rate if that coverage skips the categories — Bluetooth, push delivery, biometrics, background behavior — that emulators cannot exercise.

Time to verified defect. From the moment a test fails to the moment an engineer has a confirmed, reproducible defect with evidence attached, how long does it take? For most scripted suites this includes human triage time and is measured in hours to days. For a validated pipeline, where reproduction and analyst review happen before the failure ever reaches Slack or Jira, it should be measured in the time the run itself takes, because validation happens in parallel with execution.

Maintenance hours per release. How many engineering hours go into fixing broken tests — not fixing app bugs, fixing the tests themselves — per release cycle? This is the number that grows fastest as release velocity increases, because a faster-changing UI breaks more selectors more often. It is also the number most teams do not track explicitly, which is why it is worth starting even as a rough estimate; most teams are surprised by it once they do.

Real-device coverage. What share of your test suite actually runs on physical hardware rather than an emulator, simulator, or virtualized cloud device? This is a coverage metric, but a more honest one than raw test count, because it tells you how much of your suite can see the categories — radios, sensors, biometrics, real network conditions — that only exist on real phones.

Trust score. This one is qualitative but worth asking your own engineers directly: on a 1–5 scale, how much do you trust a red build from this suite to mean something real? Ask quarterly. A declining trend is the earliest warning sign of the other five metrics getting worse, because trust erodes before anyone formalizes the retriage rate that caused it.

None of these require new tooling to start. Retriage rate and maintenance hours can be estimated from a week of Slack threads and standup notes. Escape rate can be reconstructed from support tickets against test coverage maps. The point is not precision on day one — it is replacing "how many tests did we run" with "how much of what we reported was true," which is the question that actually predicts whether your test suite is an asset or a chore.