Case Study

How MySwimPro IoT Testing Reduced Change Failure Rate by 65%

Since 2014, MySwimPro has helped more than 1.5 million swimmers live happier and healthier lives. Today, swimmers in more than 180 countries use MySwimPro’s award-winning mobile and wearable apps. Named by Apple as the Best App of the Year, MySwimPro has also been featured in Sports Illustrated, Men’s Health, Forbes, and numerous other global publications.

MySwimPro is a personal coach that helps swimmers improve their performance and health by delivering personalized training plans, workouts, technique videos, and analytics. The MySwimPro app is available on iPhone, Android, Apple Watch, and Garmin smartwatches.

Mobot sat down with MySwimPro's VP of Engineering, Nick Newell, to discuss their experience with Mobot.

Tell me a little bit about your company, and what do you do there?

MySwimPro is a personal swim coach on your wrist.  We help swimmers achieve their health and performance goals by guiding them through personalized training plans and swim workouts.  As the VP of Engineering, I have the pleasure of leading the engineering and product team. We're continuously improving our DevOps engineering culture so we can crank out high quality software products super quickly. It's all about putting technology in the hands of our members that will empower them to reach their swimming goals. This will make them healthier and happier. And that will change their lives! So basically, I get to change lives every day. Doesn't get much better than that.

What was going on that was a problem before you started using Mobot?

Our Speed-Quality balance was a bit lopsided.  We were shipping mobile app updates rather rapidly but without sufficient regression testing. This meant that some defects were impacting the member experience.  When a swimmer is at peak performance in the final Sprint set of a 2000-meter workout, he expects the MySwimPro Coach on his Apple Watch to be at peak performance as well.

How did you hear about Mobot?

We did a pretty exhaustive search for mobile (and especially wearable) Quality Assurance applications and found surprisingly few.  Mobot immediately stood out as one of the few solutions that could help us test our wearable experience.

What convinced you to try it? 

The willingness to test our Apple Watch app alongside our iPhone app.  Swimmers don't achieve their goals on the couch or in the locker room.  They need Coach most while in the water, so this means the quality for our WatchOS app is paramount.  To help us ensure that, the team at Mobot committed to rolling up their sleeves and creating some innovative robotic technology that had never been used for wearable QA before.

How did the implementation go?

The pilot and implementation were seamless with Mobot taking on all of the heavy lifting.  We just had to communicate our expectations and they made it happen rather quickly.

What level of work effort was required by your team?

Very little.  Like I said, our effort was primarily just communicating what we wanted to test.  Mobot made it extremely easy to get started and see results.

How did you like it at first? 

Once we got off and running, we never looked back.  We started reviewing results weekly, identifying defects, and adding tests.

Did you have any trouble with it?

No trouble.  We had to identify a cadence of review and an integration with our workflow management system to maximize value, but I wouldn't really call that trouble.  It's more of an enhancement for efficiency.  

How has it helped?

It has helped us identify numerous defects within 24 hours of releasing Beta code, and this allows us several days to fix before shipping to members.  

What do you enjoy the most about our product?

The side-by-side comparison screenshots of the app, especially the recently added Apple Watch screenshots.  These paint a clear picture of the potential defect immediately.

Any success measures you can share?

Mobot helped MySwimPro to reduce our Change Failure Rate by 65% - going from 34% to 12%.

What has your team been doing with the time/budget they save?

With less time spent on triaging and reacting to Production defects, we spend more time proactively delivering new features for our members.  

Are there specific stories or anecdotes you can tell me about using the product?

Seeing that side-by-side comparison of the Apple Watch app for the first time was a game changer!  It really puts us in the water with our members when we can see the app that is on their wrist while they work hard to achieve their goals.  Only seeing the iPhone app that is in their gym bag on the pool deck is not good enough, and that is what most mobile QA solutions offer.  

What would you tell other people who were thinking of using the product?

Just get started and you'll be amazed how quickly you start seeing test results that can immediately increase your quality.  Also, don't be afraid to ask for creative solutions to your unique mobile QA problems.  The team at Mobot was incredibly friendly, open-minded, flexible, and innovative when determining the best solution for MySwimPro.  I have no doubt that they will offer the same level of service to all customers.  

5 star icon

MySwimPro members rely on us to reach their fitness goals, and we rely on Mobot to make that happen. When a swimmer is at peak performance in the final Sprint set of a 2000-meter workout, he expects the MySwimPro Coach on his Apple Watch to be at peak performance as well. To help us ensure that, the team at Mobot rolled up their sleeves and created some innovative robotic technology that had never been used for wearable QA before. They're pushing the boundaries of Mobile QA so that our members can push their fitness boundaries.

Nick Newell
VP of Engineering