Annual Defect Report 2026Download
Blog & Reports
ArticleSeptember 4, 2026 · 8 min read

Why Mobile Tests Are Flaky (and How Real Devices Help)

Flaky mobile tests usually come from timing, shared state and test data, device and OS differences, or the network. Some of that you fix in the test. Some of it only shows up on a real device. Here's how to tell which is which.

Mobile tests are flaky when the same test passes and fails without a code change. Common causes are timing (the test acts before the app is ready), shared state and test data, device and OS differences, and network conditions. Fix timing and state in your tests; find device and network failures by running on real devices.

Every team with a mobile test suite eventually has the same conversation: a test that passed yesterday failed today, nothing in the app changed, and someone has to decide whether to re-run it, quarantine it, or dig in. Multiply that across a suite and flakiness stops being an annoyance and becomes a cost: engineering hours spent re-running, investigating, and eventually ignoring results that should have been trustworthy.

There isn't one cause, so there isn't one fix. Some flakiness lives in the test and the app, and that's where you fix it. Some lives in the environment the test runs on, and no change to the script will reach it. The useful skill is telling the two apart.

Common causes of flaky mobile tests

  • Timing and synchronization. Mobile apps are asynchronous. A test that acts before the screen is ready fails some runs and passes others. Android's testing guidance calls these synchronization issues "a primary source of flakiness," and warns that fixed sleeps make tests "unnecessary slow or flaky."[A2]
  • Shared state and test data. Data left behind by an earlier run, a cached login, or an account another test already changed. Maestro's docs note that "Android apps often cache data that can lead to flaky tests," which is why it offers a command to clear app state between runs.[M1]
  • Device and OS configuration. A system update dialog or a full disk can interrupt a test. Android's guidance lists configuring devices correctly as one of the primary measures against flakiness.[A2]
  • Device and OS differences. Some failures appear only on particular hardware or OS versions. A study of flaky tests in Android apps traced 22% of the flaky commits it examined to "the usage of certain hardware, Android OS version, or a third-party library."[R1]
  • The test environment itself. In Google's 2017 analysis of its own continuous integration, Android emulator tests were the flakiest category measured.[R2] Practitioners interviewed in a 2022 study described flakiness that "stems from interactions between the system components, the testing infrastructure, and external factors," not only from how a test was written.[R3]
  • Network conditions. Slow, dropped or changing connections can make the same step pass on one run and time out on the next.

How to fix flaky mobile tests in your scripts

Start with the causes you control. Most of them are fixed the same way in any framework:

  • Wait for a condition, not a clock. Wait until a specific element, text or state is on screen instead of pausing for a fixed time. Espresso offers idling resources, and Compose offers wait-until APIs.[A2] Maestro waits for the screen to "settle" before acting.[M2]
  • Reset state between tests. Start each test from a known state, and create the test data it needs rather than relying on what an earlier test left behind.[M1]
  • Use stable selectors. Prefer accessibility identifiers and visible text over brittle paths or screen coordinates. Maestro's docs recommend text and accessibility identifiers for exactly this reason.[M3]
  • Configure test devices. Turn off system interruptions on the devices and emulators you manage.[A2]

Retries hide flakes

Retries are useful, and they are also how flaky tests go unnoticed. Android's guidance recommends retry mechanisms for large end-to-end tests, with a clear condition: "Add retrying mechanisms to big tests, but always fix flaky tests."[A2] Frameworks offer them too. Maestro, for example, has a retry command, and Maestro Cloud retries some executions automatically.[M4]

The risk is that a retry turns a real intermittent bug into a green build. Track which tests pass only on a retry, and treat that list as work to do, not noise. The same Android study found that 13% of the flaky commits it examined simply skipped or removed the flaky test rather than fixing it.[R1]

When it really is the device

Once timing, state and selectors are under control, the failures that remain are often the ones the test environment shapes. Researchers at HKUST studied 191 real-world compatibility issues in popular open-source Android apps and found that "112 (59%) of the 191 issues are device-specific," appearing only on particular device models.[R4] An emulator or simulator can't show you a defect that lives in a specific phone's hardware, drivers or OS customization.

The same goes for hardware paths a virtual device can only simulate: Bluetooth pairing with a real peripheral, push delivered to a real phone, Face ID and Touch ID against a real secure enclave, and the real camera. You find out whether those work by running on real devices. (Why real devices)

That is the case for testing on physical hardware, mechanically operated, the way Mobot does it. Mobot testing robots tap a real screen on a real phone running the real OS, and computer vision reads the screen the way a person does instead of looking up a selector. Real devices don't remove flakiness entirely: networks still vary, and apps still have real intermittent bugs. But a failure on a real device is a failure on the hardware your users hold.

Who decides a failure is real?

Even on real hardware, a failure needs a judgment call: a real defect, or a one-off blip? Every failure Mobot surfaces is reproduced in more than one environment, captured in screenshots on the physical device, and reviewed by a QA analyst before it reaches your team. (Analyst-verified defects)

The result is a suite where a red result means something, because a person has already confirmed it.

Device and platform coverage

Fragmentation is part of the flakiness problem: thousands of device models, several OS versions in active use, and different screen sizes, chipsets and vendor customizations, each with its own quirks. A few practices keep coverage manageable:

  • Pick your priority devices and OS versions from your own analytics and market share, and review the list whenever new devices and OS releases ship.
  • Use emulators and simulators for fast feedback while you build, and real devices for anything that depends on hardware, the OS or the network.
  • Build device coverage into the test plan for each release, and run compatibility passes on new OS betas before they reach your users.

Where device clouds fall short

Remote device clouds remove the cost of buying and maintaining phones, but they have limits worth knowing. The devices sit in a data center, so remote control adds latency, devices can drop offline, and the slower loop can add flakiness to automated runs. Access is usually limited to your app and some system apps, which makes it hard to test the camera, fingerprint sensor, microphone or system settings, and close to impossible to pair a smartwatch, Bluetooth accessory or other IoT device. Shared devices raise security questions for sensitive apps, and device coverage depends on the provider’s inventory and plan. Use a device cloud for what it does well, and cover hardware-dependent flows on real devices you control or through a partner that operates them physically.

FAQ

What causes flaky tests on mobile?

The most common causes are timing (the test acts before the app is ready), shared state and test data left by earlier runs, device configuration and OS differences, the test environment itself, and network conditions. Android's testing guidance calls synchronization issues "a primary source of flakiness."

How do I fix flaky mobile tests?

Fix the causes you control first. Wait for specific conditions instead of fixed sleeps, reset app state and test data between tests, use stable selectors such as accessibility identifiers, and configure your test devices. Then run on real devices to find the failures that only appear on particular hardware, OS versions or networks.

Are retries a fix for flaky tests?

No. Retries keep a pipeline moving, but they can hide real intermittent bugs. Android's guidance is to add retries to big tests but always fix flaky tests. Track which tests pass only on a retry.

Do real devices remove flakiness entirely?

No. Real devices remove the failures caused by emulators and simulators that your users never see, but networks still vary and apps still have intermittent bugs. That's why Mobot has a QA analyst review every failure before it's reported.

Can my coding agent see why a test failed?

With Mobot, yes. Every Mobot plan, including the free trial, includes read-only MCP access to test reports, test cases, observations, and device and network logs where captured. Agents read results through this access; they don't run tests or control devices through Mobot. (Read-only access for coding agents)

Keep reading

Sources

Read Oct 13, 2026. Figures are as reported by each source.

Want the full picture?

See how defect validation works
Test automationFlaky testsReal devices

More from Mobot

Guide

The Real Cost of Appium at Scale (and the Four Things That Actually Fix It)

Article

Testing Is the Bottleneck in the Agentic Software Factory

Article

What Physical Robots Catch That Emulators Miss

See What Your Emulators Are Missing

Get a real, verified defect report from Mobot’s robots and QA analysts — on your app, on real devices.

Start free trialGet a Sample Report

or talk to sales