Citizen Gets 5-Star Rating & Eliminates 1,000s of Manual Testing Hours

Who They Are

Citizen, dubbed “The most powerful safety app”, is a top-five ranked app that delivers real-time safety alerts and live video of incidents happening near their users. With apps on iOS and Android, Citizen has over 10 million users, 50% of which are active, as of January 2022. 

Citizen monitors 911 calls and keeps its users notified of crimes, accidents, and other incidents in their neighborhoods. These notifications have urged people to evacuate burning buildings, deterred school buses from nearby terrorist attacks and have even led to a rescue of a 1-year-old from a stolen car. 

Louis Johnson heads up QA for Citizen, and is responsible for testing and new features across multiple teams and projects. Louis has been in QA for over a decade, and has seen it all. He is well versed in the nuances of mobile app testing, test automation, emulators, and manual testing. 

The Challenge for Citizen

To meet the demands of their rapidly growing app and burgeoning user base, Citizen’s product and engineering teams are constantly developing new features, with a new release to the app stores once per week.

"QA is always the last hire"

Lou joined Citizen as the lone-QA, responsible for ensuring the highest quality experience for Citizen’s millions of users throughout the country. Prior to a release, Lou knew the importance of running thorough regression tests, but the priority was primarily focused around new feature testing. 

The Citizen team would execute tests using emulators with manual tests on 1-2 physical devices, but they needed much greater device coverage to ensure a seamless user experience. For Lou, device procurement was difficult due to the cost, time to manage devices, and amount of devices for full coverage.

• Rapid growth and limited QA resources led Citizen to frequently release without full device coverage. 
• Releases would lead to a negative customer experience on several device/OS combinations, ultimately causing negative reviews and user churn. 
• Testing on emulators wouldn’t catch all issues, resulting in bugs slipping through the cracks

"I've never seen an emulator get something 100% correct. When it comes to mobile it is not the same. Nothing can beat a physical device."

Mobot Solution

Mobot’s Customer Success Team worked closely with Citizen to map out a strategy addressing each of their main pain points, and a test suite specifically designed for their release process and business.

While the Citizen team is rapidly building new features, Mobot is used on a weekly basis to ensure Citizen's 5M users have access to a stable production release.

• Using a combination of Citizen user data and Mobot’s data regarding device usage, the test plans were designed to include a large number of physical devices with comprehensive coverage.
• Mobot’s team created test plans addressing critical user flows, monitoring performance of the app including battery usage, multi-device interactions (ex: messaging a friend), location spoofing, and much more.
• A shared Slack channel was set up where Citizen can monitor the execution of tests, view test results as they’re completed, and communicate priorities for the week.
• As the app has updated over the 2+ years of the partnership, Mobot’s team has updated each test plan as needed, eliminating hundreds of hours of work for Citizen.

“Mobot eliminates the smoke test. We can test in a day whereas a QA team would normally take multiple days.”

The Results

Some highlights of the 2+ year partnership:

• App store rating from a 3.9 to a 4.8
• 5x QA efficiency, measured in hours spent testing prior to release
• 4x increase in physical device coverage
• 600 test cases run each week
• Thousands of bugs caught, including many undetected by internal systems

“Feels like magic to our team”