Annual Defect Report 2026Download
Blog & Reports
GuideSeptember 5, 2026 · 14 min read

The Real Cost of Appium at Scale (and the Four Things That Actually Fix It)

Appium is free to download and much more labor-intensive than most teams realize. Where the hidden labor goes (scripts, locator fixes, device and OS upkeep, flake triage and infra), what the research says, and a model you fill in with your own numbers.

Appium is free to download, but it is far more labor-intensive than most teams budget for. Someone has to write every script, fix locators when releases change the UI, keep devices and OS versions working, triage flaky failures, and maintain the server, drivers and tooling. Model that labor with your own numbers.

What does Appium maintenance cost?

Appium itself costs nothing. It's an open-source project[AP0], and its cost is labor. Our contention is that Appium is much more labor-intensive than most teams realize, because the work is spread across five places and no single one of them looks large on its own:

  1. Script writing. Every test is code your engineers write, in a client library for their language, against Appium core, a driver per platform and optional plugins.[AP1]
  2. Locator upkeep on every release that changes the UI. Scripts find elements through locators, and published research finds most test-code changes are repairs to tests the UI broke.[R1]
  3. Device and OS matrix upkeep. Each real iPhone needs WebDriverAgent installed with a valid provisioning profile, Developer Mode and Enable UI Automation[AP3], and the iOS chain runs through Xcode, iOS and macOS, which all move on their own schedules.[AP2]
  4. Flake triage. Someone has to decide whether each red result is a bug.[R5][R6]
  5. Infra. The Appium server, drivers, client libraries, Node.js, SDKs, Xcode and the devices or device cloud all have to keep working together. Appium's docs call the iOS version of that chain "a pretty deep stack."[AP2]

Each of those is somebody's time, every sprint. There's no single industry number for it, and you should be wary of anyone who gives you one (see the research section below). The rest of this post breaks the labor into parts you can measure, and ends with a model you fill in with your own numbers.

Appium is a good tool. The labor is the bill.

Appium is free, open-source, and genuinely capable — it remains the most widely used cross-platform mobile automation framework, and none of that is in question. What's worth examining is what "free" actually costs once a team scales past a few dozen tests on a fast-moving app. The license fee is zero, and the labor isn't. The gap between the two is not obvious until a team is deep enough in to feel it, because the labor is spread across people, tickets and sprints and rarely shows up as one line item.

This is not an argument that Appium is a bad choice. It's a breakdown of where the labor goes, where the cost curve stays flat, where it bends, and what actually changes its shape — so the decision can be made on the real cost, not on the license price alone.

Where the hidden labor goes

"Test maintenance" is not one line item. It's at least five, and most teams only budget for the first one.

1. Script writing

Every Appium test is code. Your engineers write it in the client library for their language (JavaScript, Java, Python and others), against Appium core and a driver for each platform.[AP1] Writing a test is only the start: it also has to be stabilized, wired into CI and reviewed.

Appium's core promise is write-once-run-both across Android and iOS. In practice, platform-specific quirks — different accessibility label conventions, different timing behavior, different gesture handling — creep into the "shared" suite over time, and teams end up maintaining meaningfully divergent test logic per platform anyway. The promise is real but partial: a team still absorbs some of the duplication cost the shared framework was supposed to eliminate.

2. Locator upkeep on every release that changes the UI

This is the visible labor: a test fails, someone opens it, the locator no longer matches the element it was written against, and they fix it. It's also, structurally, unavoidable at scale, because Appium (like any locator-based framework) ties a test's reliability to a specific attribute — a resource ID, an XPath, an accessibility label — and modern mobile UIs do not hold those attributes still. A renamed view, a restructured layout, a text label updated for a new locale: any of it invalidates every test that located an element through the changed attribute.

The research backs up how much of the work this is. A study of Android GUI test suites found test code "has to be modified often," and that about 60% of those modifications "are induced by fragility."[R1] Uber's mobile testing engineers reported spending "30–40% of their time on maintenance."[R3]

3. Device and OS matrix upkeep

Real-device coverage is labor too. To run on a real iPhone, the XCUITest driver has to install WebDriverAgent on the device, which needs a valid provisioning profile, Developer Mode turned on, and Enable UI Automation turned on.[AP3] Appium's docs say managing WebDriverAgent "can be a pain and involves the use of Xcode."[AP2] On iOS, the chain of technologies includes Xcode, XCUITest, iOS and macOS[AP2], and each of those versions moves on its own schedule. On Android, the UiAutomator2 driver relies on ADB and the Android SDK as well as UiAutomator2 itself.[AP2]

The matrix also matters for what you find. An Android-specific study traced 22% of flaky commits to "the usage of certain hardware, Android OS version, or a third-party library"[R7], and researchers at HKUST found that 59% of 191 real-world compatibility issues were device-specific.[R8] Covering more devices and OS versions finds more of those issues, and every device and version you add is another thing your team has to keep working.

4. Flake triage

Every red result needs a person to decide whether it's a bug. Slack's engineering team reported that "each test failure takes about 28 minutes to manually triage."[R5] A study of Chromium's continuous integration found that "false alerts represent more than 81% of the failures detected."[R6] In Google's 2017 analysis, Android emulator tests were the flakiest category measured.[R4] Triage labor scales with how many failures your suite produces, not with how many bugs your app has.

5. Infra

Before a single test runs, someone has stood up Node.js, the JDK, the Android SDK, platform drivers, and environment variables — and kept them current every time a driver or OS version moves. Getting started with Appium means installing Appium itself, a driver per platform, a client library and, optionally, plugins[AP1], and those parts have to stay compatible with each other. Appium's docs describe the iOS chain, from your test code and client libraries through the Appium server, the XCUITest driver, WebDriverAgent, Xcode and iOS, as "a pretty deep stack," where "it can be difficult sometimes to tell where in this chain of technologies something has gone wrong."[AP2] Then there are the devices or device cloud, the CI jobs, and the results storage. None of this is instantaneous, and almost none of it is optional if the suite needs to keep running.

What the labor crowds out: the coverage that never gets written

This is the cost that never shows up in a maintenance audit, because it's an absence, not a logged event. When most of a QA team's time goes to keeping existing tests alive, the capacity left for writing new coverage shrinks — and teams quietly stop automating the parts of the app they know should be tested, because there's no time left. It never appears as a line item. It's still a real cost, and it's usually the largest one.

Running Appium on Firebase Test Lab or a device cloud

Teams that don't want to run their own device lab usually look at a hosted option. Two things are worth knowing before you plan around one.

  • Firebase Test Lab doesn't list Appium as a supported framework. Asked whether Test Lab supports Appium, Firebase's FAQ says: "we're currently unable to provide commitment to supporting these testing and app development platforms."[F1] Test Lab runs Android instrumentation tests (Espresso or UI Automator), Robo tests, Game Loop tests and XCTest for iOS[F2]. Firebase also says Test Lab "is deprecated and will be fully shut down on September 30, 2027," with migration to the Developer Device Platform on Google Cloud.[F1] Its pricing model is a no-cost daily testing quota, with billing by the minutes spent running tests above that on the Blaze plan.[F3]
  • Device clouds that run Appium include Sauce Labs[S1] and BrowserStack App Automate[B1]. They give your team real and virtual devices to run its own scripts on, and some document hardware features for Appium tests. BrowserStack, for example, documents camera image injection and biometric authentication, and lists both as available only on certain plans.[B1][B2] Pricing is usually by plan, so check each vendor's pricing page against your suite size and how many runs you need in parallel.

Either way, a hosted device removes the device lab from your plate. It doesn't remove script writing, locator repair or failure triage. Those stay with your team.

What the data actually says — and what it doesn't

A figure circulating widely in 2026 claims teams running 200-plus Appium tests spend 60–70% of QA engineering time fixing broken selectors. It gets repeated as though it were a research finding. It originates in the marketing content of Drizz, a vendor selling an Appium alternative. On most of the pages where it appears it carries no attribution at all; where it is attributed, it points to an unpublished internal "Analysis of 40 Startups, Q4 2025" with no methodology, sample description, or document a reader can inspect. We could find no independent research behind it, and no use of the figure predating Drizz's. So we won't cite it — and we'd encourage you to check the provenance of any number a testing vendor quotes you, ours included.

Here is what is actually documented, by researchers and engineering teams with nothing to sell you:

  • About 60% of mobile test maintenance is repairing what the UI broke. Coppola, Morisio and Torchiano at Politecnico di Torino studied Android GUI test suites and found that test code "has to be modified often," and that "a relevant portion (60% on average) of such modifications are induced by fragility" — repair work caused by interface change rather than new coverage. Peer-reviewed, Android-specific, and the properly sourced version of the number the vendor blogs invented.
  • What that costs in practice. Slack's engineering team — 16,000+ automated Android tests, 11,000+ on iOS, 550+ pull requests a week — reported that 57% of their CI failures "failed due to test job failures consisting of flaky and failing automated tests," and that "each test failure takes about 28 minutes to manually triage." Fixing it recovered 553 hours of triage time. Mobile-specific, operational, and from a company that sells chat.
  • A named engineering team's own number. Uber's DragonCrawl team wrote that engineers working on mobile testing "invest 30–40% of their time on maintenance," with tests "not immune to frequent disruptions caused by minor updates, such as new pop-ups and changes in buttons." That is lower than the vendor figure — which is rather the point.
  • Even Google — and emulators are its flakiest rung. Google reported in 2017 that of roughly 4.2 million continuous-integration tests, about 63,000 (some 1.5%) had a flaky run in a given week. In the same post's per-tool breakdown, Android emulator tests were the flakiest category measured, at 25.46%.
  • Most CI failures aren't defects at all. A study of Chromium's continuous integration (an arXiv preprint, so not peer-reviewed) analysed more than a million test failures across 2,000 builds and found that "false alerts represent more than 81% of the failures detected in the Chromium CI, whereas legitimate failures only represent 19%." That is a desktop browser project rather than a mobile app — but the ratio is the signal-to-noise problem in a single line.
  • And flakiness is structural, not a discipline problem. Interviews with practitioners found that flakiness "stems from interactions between the system components, the testing infrastructure, and external factors," not merely from carelessly written tests. An Android-specific study put 22% of flaky commits down to "the usage of certain hardware, Android OS version, or a third-party library" — and found 13% of commits simply skipped or removed the flaky test rather than fixing it.

One thing we looked for and could not find: a credible industry survey of mobile test maintenance cost, with published methodology, from a publisher that is not selling the solution its findings imply. That space is occupied almost entirely by testing vendors. We would rather tell you that than dress one up.

For what flaky failures look like and how to fix them, see Why mobile tests are flaky (and how real devices help). For why the cost compounds over time, see Your test automation creates costly test debt.

When the cost curve stays flat

It would be inaccurate to frame this as Appium being broadly wrong for mobile QA. The cost above bends under a specific set of conditions — outside them, Appium remains a reasonable choice.

The cost stays manageable when the UI is relatively stable between releases, the team already has dedicated automation engineering capacity in headcount, the suite stays small, or the team needs deep custom control — proprietary hardware integrations, non-standard reporting — that Appium's open plugin architecture supports well.

It bends when the UI changes weekly or faster, when QA generalists rather than dedicated SDETs are asked to maintain the test code, or when release cadence is fast enough that a maintenance backlog directly delays shipping. Most teams evaluating this honestly find themselves in the second set by the time they're asking "why is our suite always red" — which is a signal in itself.

What actually changes each category

"Add AI" is not specific enough to be useful. Here is what structurally addresses each kind of labor — and why a fix aimed at one does not automatically fix the others.

Locator upkeep: remove the selector, not just make it smarter

AI locator repair tools — building a composite match from text, position, and surrounding structure instead of one attribute — reduce the cost of locator upkeep, but they're still repairing the same underlying mechanism: a script that has to find an element to act on it. Mobot testing robots don't locate elements at all. Computer vision reads the screen the way a person does and a robot physically taps the device, so there is no selector, repaired or otherwise, to break when a button is renamed or a screen is redesigned. This doesn't reduce the cost of locator upkeep. It removes the mechanism that generates it.

Script writing: nothing shared, nothing to diverge

Script writing and its cross-platform duplication come from maintaining code meant to describe two platforms that don't behave identically. With Mobot, Mobot's team and Mobot AI build the test cases, and Mobot's AI-assisted authoring generates coverage directly from each platform's actual build — the iOS coverage is authored against the iOS build, the Android coverage against the Android build, each executed by robots on real hardware for that platform. There is no shared script layer to quietly diverge, because nothing was shared to begin with.

Infra and the device matrix: it's not on your plate

There is no SDK to install, no driver to version-bump, no WebDriverAgent to provision and no device farm to configure, because Mobot owns and operates the device lab — 300+ real iOS and Android phones and tablets. A build goes in; robots run it. The remaining integration work — connecting results to Slack, Jira, or TestRail — is a one-time setup, not an ongoing tax.

Flake triage and the unwritten-coverage gap: capacity, not cleverness

These aren't fixed by a smarter tool — they're fixed by having dedicated capacity that doesn't compete with your engineers' other work. Every failure Mobot surfaces is reproduced in more than one environment, captured in screenshots on the physical device, and reviewed by a QA analyst before it reaches your team, so triage isn't your engineers' job. AI-assisted authoring proposes new coverage as your build changes, so coverage grows with the app instead of trailing a backlog. And because Mobot runs as a service with its own fleet and its own QA analysts, the constraint that produces the coverage gap in the first place — "we didn't have the hours" — doesn't apply the same way; capacity scales with the plan, not with how much is left over after keeping the existing suite alive.

Another category: what virtual devices can't exercise

Every kind of labor above assumes the test could technically run — the only question was whether it broke. There's a separate category of cost that no selector strategy, with or without automated locator repair, changes: what a simulator or emulator can exercise at all. Bluetooth pairing with a real peripheral, push delivered to a real phone, Face ID and Touch ID against a real secure enclave, and real camera input don't exist on an emulator or simulator. Appium can reach some of them on real devices or device clouds, which document camera and biometric support[B1][B2], but someone still has to script, maintain and triage every one of those tests.

This is measurable, and it has been measured. Researchers at HKUST examined 191 real-world compatibility issues from popular open-source Android apps and found that "112 (59%) of the 191 issues are device-specific" — defects that appear only on particular device models, traced to problematic driver implementations, OS customizations, and hardware composition. An emulator cannot surface those by definition; there is no driver and no silicon to be wrong. A 2024 follow-up study found the same pattern still holds across Samsung, Xiaomi, Huawei, OPPO, Sony, LG and Vivo builds.

That gap has nothing to do with how well-maintained the suite is. It's closed by testing on real hardware, or it isn't closed.

Real examples

Homebase, the workforce management platform used by 150,000+ small businesses, had tried traditional UI automation multiple times but kept falling back to manual testing — the suite was too expensive to keep alive against a platform with that much surface area. Regression only completed once a quarter on Android, less often on iOS, against a release cycle shipping every two weeks. After moving to Mobot, the team automated 100+ end-to-end test cases per platform in under four months and brought full-coverage regression time down from 5–10 days to same-day results.

Read the full Homebase case study →

A Canadian real-estate platform had a similar story. It tried open-source automation frameworks first, and found that tests "frequently broke due to minor UI changes." It moved its mobile app testing to Mobot, and Mobot's team now handles test case development, execution and maintenance.

Read the real-estate platform case study →

Calculate your own number: a hidden-labor model

The labor above is the thing worth measuring before deciding whether the fix is process discipline, a tooling change, or a different testing model entirely. The model below has no default values and no Mobot results. Every input is a number you set from your own team. Track each one for a sprint or two before you trust it — repair hours and new-coverage hours get conflated constantly, and conflating them is the single biggest reason teams underestimate their own maintenance cost.

Inputs (all set by you):

Input What it means
N New tests written per month
Hₐ Hours to write, stabilize and wire one new test into CI
S Number of end-to-end tests in your suite
K Releases per year that change the UI
P Share of the suite needing a locator or script fix after one of those releases
Hᵣ Hours to diagnose, fix, re-run and review one broken test
D Device and OS combinations you keep working (real devices, simulators, emulators or cloud configurations)
U Update events per year that touch those combinations (OS, Xcode, Appium, driver or device changes)
Hd Hours per combination per update event
F Failures per week that turn out not to be bugs
M Minutes to triage one of those failures
I Hours per month spent on infra: Appium server, drivers, client libraries, Node.js, SDKs, Xcode, CI jobs, device lab or device cloud setup
Rate Your loaded hourly engineering cost

Annual hidden labor (hours):

  1. Script writing = N × Hₐ × 12
  2. Locator upkeep = S × P × Hᵣ × K
  3. Device and OS matrix upkeep = D × U × Hd
  4. Flake triage = F × (M ÷ 60) × 52
  5. Infra = I × 12

Total hours = 1 + 2 + 3 + 4 + 5. Annual labor cost = total hours × Rate.

Assumptions in this model (stated so you can change them):

  • A1. You set every input. The model ships with no defaults. Any number you see in a worked example is the reader's own, not ours.
  • A2. The five kinds of labor don't overlap. If one hour of work covers two of them (for example, an OS update that also breaks locators), count it once.
  • A3. Each category scales linearly with its inputs. Real suites can be better (shared fixes) or worse (cascading breakages) than linear.
  • A4. Locator upkeep is counted only for releases that change the UI (K), not every build.
  • A5. Matrix upkeep is counted per combination, per update event. "Update event" covers OS releases, Xcode releases, Appium and driver updates, and device changes. All of these are parts of the chain Appium's docs describe[AP1][AP2][AP3].
  • A6. Flake triage counts only failures that turn out not to be bugs (F). Time spent fixing real defects isn't hidden labor and is excluded.
  • A7. Infra is a monthly average (I). Spikes, such as a major Appium or Xcode upgrade, are averaged over the year.
  • A8. Excluded on purpose: device and device-cloud fees, CI compute, the cost of defects that escape to production, and the coverage that never gets written. Price these separately if they matter to you.
  • A9. Published figures are sanity checks, not defaults. If your total is far from what others report, recheck your inputs. Slack reported about 28 minutes to manually triage each test failure[R5]. Uber reported mobile testing engineers spending 30–40% of their time on maintenance[R3]. A study of Android GUI suites found that, on average, about 60% of changes to GUI test code were caused by fragility[R1]. These come from different teams and suites, so treat them as ranges to compare against, not inputs.
  • A10. To compare against the Uber figure, divide your total hours by (number of engineers on the suite × the productive hours each works per year). That gives the share of their time going to hidden labor. Both of those numbers are yours.

An interactive version of the model is on the Mobot vs. Appium page.

Moving to Mobot? We'll convert your Appium or Maestro tests for free. Share your existing Appium or Maestro tests. Mobot's team turns them into Mobot test cases, which are resilient by nature, as part of onboarding. This also applies during the free trial. Start a free trial (1,000 free Mobot credits) or talk to our team.

How test debt builds up

Test debt is the maintenance, triage and rework a suite needs just to keep telling you the truth. It isn’t a sign of a bad team. It comes from describing a user interface in code: every selector is a promise that the screen won’t change, and every wait is a guess about the environment. It builds up in three ways:

  • Authoring debt: new features need new scripts before they’re covered, so coverage lags the roadmap.
  • Maintenance debt: existing scripts break with refactors and UI changes, and the repair backlog grows as you ship faster.
  • Trust debt: as flaky failures pile up, teams stop believing red builds, and the suite stops doing its one job.

FAQ

Is Appium free?

Yes. Appium is an open-source project and free to download. The cost of using it is engineering time: writing and maintaining scripts, keeping Appium, its drivers and your tooling current, and triaging failures, plus whatever devices or device cloud you run on.

What is the cost of maintaining a mobile test automation suite?

It depends on suite size, how often your UI changes and who maintains the suite. Published figures vary widely: Uber's mobile testing engineers reported 30–40% of their time going to maintenance, and a study of Android GUI test suites found that, on average, about 60% of changes to GUI test code were caused by fragility. Track your own repair, authoring and triage hours to get your number.

Is Appium's maintenance cost really that high?

It depends entirely on suite size and how fast the UI changes, and the honest published range is wide: Uber's mobile testing engineers reported 30–40% of their time going to maintenance, while academic work on Android GUI suites found that, on average, about 60% of changes to GUI test code were caused by fragility rather than new coverage. A small, stable suite maintained by dedicated automation engineers can stay cheap for years. A large suite on a fast-shipping app, maintained by QA generalists between other responsibilities, is where the line grows fastest. Measure your own number rather than trusting anyone's headline figure, including this one.

Do AI locator repair tools solve the problem?

They meaningfully help with selector repair specifically, because that's the exact mechanism they target. They do not touch cross-platform duplication, setup overhead, or the coverage that never gets written, because those come from different causes and need different fixes.

When does it make sense to stay on Appium?

When the suite is small, the UI is stable release to release, dedicated automation headcount already exists regardless of tooling choice, or the team needs a level of custom integration control that a managed platform doesn't offer. The cost curve genuinely stays flat under those conditions.

What's the single biggest hidden cost teams miss?

The coverage that never gets written. It never shows up in a maintenance-hours audit, because it's an absence rather than a logged event — but it's a real cost, and it's usually the one worth pricing first.

Keep reading

Sources

Appium and testing platforms (official docs, read Oct 13, 2026):

Peer-reviewed and academic:

Engineering teams reporting on their own systems:

Figures are as reported by each source and linked so you can check them. Where a source sells testing tooling, we've said so in the text. Mobot's own numbers come from the Homebase case study and the real-estate platform case study.

Want the full picture?

See the Mobot vs. Appium comparison
Test automationTest debtAppium

More from Mobot

Article

Why Mobile Tests Are Flaky (and How Real Devices Help)

Article

Why Black Friday Promo Links Break After the Tap (and How to Catch It Before Launch)

Article

Testing Is the Bottleneck in the Agentic Software Factory

See What Your Emulators Are Missing

Get a real, verified defect report from Mobot’s robots and QA analysts — on your app, on real devices.

Start free trialGet a Sample Report

or talk to sales