Flaky Tests Are a Device Problem, Not a Script Problem
Most teams treat flakiness as a scripting problem and respond with retries, waits, and self-healing selectors. That treats the symptom. The underlying cause for most mobile flakiness is that the test never touched a real device in the first place.
Every team with a mobile test suite eventually has the same conversation: a test that passed yesterday failed today, nothing in the app changed, and someone has to decide whether to re-run it, quarantine it, or dig in. Multiply that by a few hundred tests a week and flakiness stops being an annoyance and becomes a line item — engineering hours spent re-running, investigating, and eventually ignoring results that should have been trustworthy.
The standard response is to treat flakiness as a scripting problem. Add retries. Lengthen waits. Adopt a framework with self-healing selectors that finds the button even after the UI shifts. These fixes help at the margin, and they are also treating the symptom. A script can retry a step, but it cannot retry its way into a fact it never established: did this happen on the hardware a user actually holds?
Most mobile test suites run against an emulator, a simulator, or a software-driven cloud device — a virtualized rendering of an OS, not the OS itself. That layer of virtualization is where a large share of flakiness actually originates. A simulator schedules CPU and network differently than a physical phone under thermal load. It has no real radio, so Bluetooth and cellular tests are stubbed or skipped, and the parts that are stubbed can produce results that vary run to run for reasons that have nothing to do with your app. A cloud device farm is closer to real hardware, but it is shared, rate-limited, and running someone else's prior test's residue half the time — a different, noisier source of nondeterminism.
None of this means engineers are bad at writing tests. It means the environment underneath the test was never as stable as the test assumed. A selector retry cannot fix a race condition introduced by an emulator scheduling a frame differently than the phone will. A self-healing locator cannot fix a Bluetooth pairing that the simulator never attempted for real. The fix has to happen a layer down, in what the test actually runs on.
This is the case for testing on physical hardware, mechanically operated, the way Mobot does it. A robot taps a real screen on a real phone running the real OS, with a real radio and a real thermal envelope. There is no selector to retry, because computer vision reads the screen the way a person does, and there is no virtualization layer introducing behavior your users will never see. Some flakiness remains — networks are still variable, apps still have real intermittent bugs — but the largest category of flaky signal, the kind caused by the environment rather than the app, goes away because the environment is no longer synthetic.
The second half of the fix is validation. Even on real hardware, an occasional failure needs a human judgment call: is this a real defect, or a one-off environmental blip? That is why every failure Mobot surfaces is reproduced in more than one environment and reviewed by a QA analyst before it reaches your team. A test that fails once and never again is recorded, not reported, until it either reproduces or resolves. Your Slack channel only hears about the failures worth acting on.
The result is a suite where a red result means something. Teams stop budgeting "flaky test triage" time into every sprint, because the category of failure that used to eat that time — the one caused by the simulator, not the app — has nowhere to come from anymore. That is the actual fix for flakiness: not a smarter retry, but a real phone and a human in the loop.
Want the full picture?
See how defect validation works