Citizen Gets 5-Star Rating & Eliminates 1,000s of Manual Testing Hours

The Challenge for Citizen

To meet the demands of their rapidly growing app and burgeoning user base, Citizen’s product and engineering teams are constantly developing new features, with a new release to the app stores once per week.

"QA is always the last hire"

Lou joined Citizen as the lone-QA, responsible for ensuring the highest quality experience for Citizen’s millions of users throughout the country. Prior to a release, Lou knew the importance of running thorough regression tests, but the priority was primarily focused around new feature testing. 

The Citizen team would execute tests using emulators with manual tests on 1-2 physical devices, but they needed much greater device coverage to ensure a seamless user experience. For Lou, device procurement was difficult due to the cost, time to manage devices, and amount of devices for full coverage.

• Rapid growth and limited QA resources led Citizen to frequently release without full device coverage. 
• Releases would lead to a negative customer experience on several device/OS combinations, ultimately causing negative reviews and user churn. 
• Testing on emulators wouldn’t catch all issues, resulting in bugs slipping through the cracks

"I've never seen an emulator get something 100% correct. When it comes to mobile it is not the same. Nothing can beat a physical device."

Mobot Solution

Mobot’s Customer Success Team worked closely with Citizen to map out a strategy addressing each of their main pain points, and a test suite specifically designed for their release process and business.

While the Citizen team is rapidly building new features, Mobot is used on a weekly basis to ensure Citizen's 5M users have access to a stable production release.

• Using a combination of Citizen user data and Mobot’s data regarding device usage, the test plans were designed to include a large number of physical devices with comprehensive coverage.
• Mobot’s team created test plans addressing critical user flows, monitoring performance of the app including battery usage, multi-device interactions (ex: messaging a friend), location spoofing, and much more.
• A shared Slack channel was set up where Citizen can monitor the execution of tests, view test results as they’re completed, and communicate priorities for the week.
• As the app has updated over the 2+ years of the partnership, Mobot’s team has updated each test plan as needed, eliminating hundreds of hours of work for Citizen.

“Mobot eliminates the smoke test. We can test in a day whereas a QA team would normally take multiple days.”

The Results

Some highlights of the 2+ year partnership:

• App store rating from a 3.9 to a 4.8
• 5x QA efficiency, measured in hours spent testing prior to release
• 4x increase in physical device coverage
• 600 test cases run each week
• Thousands of bugs caught, including many undetected by internal systems

“Feels like magic to our team”