Platform · Defect validation
A failed test is not a bug. A verified defect is.
Automation suites and simulators report every failed assertion and leave your engineers to work out which ones matter. Mobot validates each failure before it reaches you: the forensic data is reviewed, the defect is reproduced in more than one environment, the occurrence is on video, and a QA analyst makes the call. The result is a superior signal-to-noise ratio, and a backlog your team believes.
- Forensic review
Device log, network log, and per-step screenshots are read against the failure timestamp.
device.log 00:04.2HTTP 500 /tasks3 screenshots - Reproduction
The failing step is re-run on a second device, OS version, and app state.
iPhone 13 · iOS 27 ✓Pixel 8 · Android 16 ✓cold start ✓ - Video evidence
Recorded on the physical device as the failure happens, with the robot’s tap in frame.
▶ 00:03.8 – 00:05.1 - Analyst verdict
A QA analyst reads the evidence and decides: real defect, flaky run, or intended change.
Confirmed · P0 · crash on tap
Signal-to-noise
Your automation suite is not short on failures. It is short on verdicts.
A red build from a scripted suite means one of five things, and only one of them is a bug in your app. Every other cause costs an engineer the same triage time and delivers nothing.
- noiseA locator or selector broke
The UI changed. The app works. The script does not.
- noiseA timing wait was too short
The screen loaded 400 ms late on a shared cloud device.
- noiseThe environment failed
Simulator restart, expired session, cloud device unavailable.
- noiseThe change was intended
Copy, layout, or a redesigned flow nobody updated the test for.
- signalYour app has a defect
The one your engineers need to see. The one that is hardest to find in the pile.
The retriage tax
When four out of five red results are noise, engineers learn to distrust the fifth. Triage becomes a chore that gets deferred, then batched, then skipped. The suite is still running. It has just stopped changing what anyone does.
Validation is the fix. Not fewer tests, and not fewer failures on the robots, but a filter between the failure and your backlog that only a confirmed defect can pass. Your engineers should never be the first people to ask whether a failure is real.
What validation means
Four things we do that a failed assertion never will
Each one removes a category of noise. Together they are the reason a Mobot defect report goes straight to an engineer instead of back into a triage queue.
We review all of the forensic data
Network logs, device logs, screenshots, and timing, read together.
A failed assertion tells you a screen did not match. It does not tell you why. Our analysts read the device log and the network log against the failure timestamp, alongside the screenshot at every step, to establish what actually happened: a crash, a 500 from your API, a race on cold start, or a screen that simply loaded late.
- Device log at the failure timestamp
- Network calls and response codes
- Per-step screenshots and the diff against baseline
We recreate the defect in multiple environments
Different device, different OS version, different app state.
Before a defect is reported, it is re-run. On the same device to rule out a one-off, on a second device or OS version to establish the blast radius, and from a different app state where it matters: cold start versus backgrounded, cellular versus Wi-Fi, logged in versus fresh install. What you receive says where it reproduces and where it does not.
- Re-run on the original device
- Reproduced on a second device or OS
- App state and network conditions varied
We provide video of the actual defect occurrence
Recorded on the physical device as it happens.
Every report includes video captured from the real device at the moment of failure, with the robot’s action in frame. There is no arguing about what the tester saw or whether the screenshot was taken too early. Your engineer watches the tap, watches the app respond, and sees the defect before opening a single file.
- Physical device, not screen capture from a simulator
- The action and the response in one clip
- Attached to the step where it happened
We keep a human in the loop
A QA analyst reads the evidence and makes the call.
Automation is good at noticing that something changed. It is bad at knowing whether that matters. A Mobot QA analyst reviews every failure, interprets the data, and decides: real defect, flaky run, or intended change. Only confirmed defects reach you, with the analyst’s note explaining what they found and how confident they are.
- Every failure reviewed before it is reported
- Intended changes decided with you once, then never re-reported
- Analyst note attached to every defect
What lands in your backlog
The same failure, reported two ways
Simulators and scripted suites are built to run tests. They are not built to decide what a failure means. That work is left to you.
Simulators & automation suites Appium · Maestro · XCUITest · cloud device farms | Mobot real devices · validated · human-verified | |
|---|---|---|
| What triggers a report | Any assertion failure, including timing, selectors, and environment | A defect an analyst has confirmed |
| Evidence attached | A stack trace and, sometimes, a screenshot | Video, device log, network log, per-step screenshots, repro steps |
| Reproduced before reporting | No. Retry is your CI’s job | Yes, on the original device and at least one more environment |
| Where it ran | Simulator, emulator, or a software-driven cloud device | A physical phone, operated mechanically by a robot |
| Who decides it is real | The engineer who picks up the red build | A Mobot QA analyst, before your team sees it |
| First thing your engineer does | Figure out whether it is a bug | Fix it |
Every Mobot report is built from a real run on a physical device. Open the sanitized sample to see the observation, the analyst note, and the grouped steps exactly as your team would receive them.
Open the sample report
Why real devices matter here
Validation only works when the evidence is real
Reproducing a crash on a simulator proves the simulator crashes. Reproducing it on an iPhone 13 running a developer beta, then on a Pixel on a carrier network, proves your users will hit it. Because Mobot’s robots operate physical devices, every log, every screenshot, and every second of video comes from the hardware your customers actually hold.
That is also why the analyst’s verdict carries weight. They are not interpreting a model of a phone. They are watching the phone.
Questions
Defect validation, in practice
How long does validation add before I hear about a defect?+
Regression runs execute overnight and results are validated by morning. Analysts work the failure queue as the run completes, so validation happens in parallel with execution, not after it. For a P0 crash you are notified as soon as it is confirmed.
What happens to a failure the analyst decides is not a defect?+
It is recorded in the platform with the analyst’s reason and does not reach your backlog. If it is an intended change, the baseline is updated with you once so it is never re-flagged. If it is flaky, it is tracked so a genuinely intermittent defect can still be caught over repeated runs.
Can I see the failures that were rejected as noise?+
Yes. Every run, every step, and every analyst decision is visible in the Mobot platform. Validation removes noise from what gets pushed to Slack and Jira. It does not hide anything from you.
Can my own automation suite do this?+
Partially. You can add retries, capture screenshots, and pull logs from a simulator. What you cannot get from a simulator is the real device behaviour, and what you cannot get from a script is a judgement about whether a change is a defect. That judgement is the step that turns a red build into a ticket an engineer will trust.
Get a verified defect report on your app
Send us a build. Robots run it on real devices, analysts validate every failure, and you get back defects your engineers can fix without a second look.