5 Hidden Weaknesses of Mobile Testing with Devices in the Cloud
A mobile device has become our daily partner. No matter which digital task has to be performed, the device is with us, helping us to solve the problem. Modern mobile devices are packed with powerful hardware running all kinds of mobile apps, and it seems like there is no end when it comes to new innovations and solutions. But, with this never ending innovation cycle comes an equal amount of challenges for mobile development teams and mobile testers.
Mobile Apps Are Like CD-ROMs
“Why did I miss this issue?” or “How is this even possible?” - questions mobile testers may have asked themselves while reading customer feedback in an app store. We all know that software isn’t 100% bug free, but the feeling of missing a bug will keep you up at night.
One of the (many) challenges with mobile app development is how to handle the situation where an app is released and contains a critical bug. Once it is on the customers’ device, rolling the app version back can be quite challenging.
It’s like in the good old days, when you bought a computer magazine with a CD-ROM. If the CD-ROM was broken, it wasn’t possible for the publisher to send a fixed CD-ROM to the customers.
Today, mobile apps are similar. If the development team has no feature toggles implemented to turn off the buggy feature, it might take days or weeks to get a hotfix out and update to the customer. A bad scenario with potential for negative user experience, reviews, crashes, and who knows what else.
To avoid such cases, mobile developers and testers have to cover a plethora of hardware and software combinations of mobile devices & operating systems. However, buying all the different devices with the different hard- and software combinations is not something I would recommend based on experience. It can be quite expensive and time consuming to procure and maintain the devices. Keeping them in the same state needed for development and testing is an exhausting task - I know!
To solve these challenges, some companies turn to mobile device clouds, with the goal of outsourcing the device selection and maintenance process. There are many different mobile device cloud providers on the market which offer a great set of features to cover the challenge of mobile device fragmentation.
Mobile Device Clouds Have Hidden Weaknesses
However, mobile device clouds have hidden weaknesses every mobile development team may not be aware of. This article describes the 5 hidden weaknesses of mobile device clouds.
1. Network Latency
One hidden weakness of mobile device clouds is network latency. The selected mobile devices are hosted by the cloud provider in a data center, and connected to the Internet. The data center may be located in a different geographical area than the development team, which will cause slower reaction times when sending the request to the device and waiting for the response to come back. The delay may only be a few milliseconds, but the latency will be tangible. This might impact the development and testing results and might confuse the development team, because the app might not feel fast and responsive to use while navigating the app via a browser with a computer mouse. In some cases, the cloud device may go offline unexpectedly due to connectivity issues - a real pain while testing. Last but not least, the slower connection may also impact the test automation results, ending up in additional unexpected test flakiness.
Therefore, I highly recommend testing your mobile app on a real device in the real hands of a mobile tester before submitting it to the app stores. Only this will truly replicate the real user experience during testing.
2. Lack of Local Access and Connection to Other Devices
One of the large issues is the connected mobile cloud devices are usually limited in access rights. This means a mobile development team only has access to the installed app and to some extent the system apps. This is a security measure added by the cloud vendor to keep the device in a clean and maintainable state. However, without having access to the physical device, it can be difficult to troubleshoot issues. In some instances, it will limit what you can test if the app requires access to some specific system apps or hardware features such as fingerprint scanner, camera or the microphone.
Another big pain point with cloud devices is that it is almost impossible to connect to another device such as a smartwatch, or other IoT device. The cloud device can only be used in isolation.
If you need multi-device testing, mobile cloud providers are not the best choice.
3. Security Concerns
If a mobile app is handling sensitive data such as user information, health records, or credit card data, security is critical. Every mobile development team must do a security risk assessment to determine if the usage of a mobile device cloud could be a potential threat.
Mobile device cloud providers usually offer a great set of security measures. However, it’s not 100% guaranteed that the devices, the data centers and the APIs are secure enough for potential security leaks or hacks.
4. Mobile Device Subscription Plans
As mentioned earlier in this article, procuring and maintaining the large amount of required mobile device and OS combinations for an internal device lab is challenging - and expensive! However, before selecting a device cloud provider, every team should take a close look at the different pricing options and subscription plans, and how it stacks against your device coverage needs. Depending on your requirements, the costs can increase significantly and quickly. Running some manual tests on a mobile device cloud won’t add to much, but running full test automation suites across the connected devices can lead to immense costs. Some cloud providers charge per-minute or per-hour, while others charge per test executed. You’ll need to assess which automated tests can be run locally, which in the cloud, etc.
5. Limited Device Coverage
Depending on the provider you choose, the range of available mobile devices may be limited. Before choosing a cloud provider, you should review usage data to determine the specific device and OS combinations needed for sufficient coverage, and whether they can meet your needs. Keep in mind, this isn’t a one-time exercise. You should update your device matrix and needs on a regular basis to ensure you have adequate coverage based on changing user behavior. Operating system versions are one that tends to change often, and some cloud providers do not support older operating system versions.
Just a piece of the puzzle
There are certainly areas where mobile device clouds can be beneficial during the testing process - they can be a powerful tool, That said, they’re simply a piece of the puzzle. As shown in this article, there are a variety of weaknesses that many mobile teams are normally not aware of. It is critical to understand these pros and cons, and carefully assess the solutions as you build your mobile testing toolkit.
And last but not least, a team must understand that relying completely on a mobile device cloud can never replace testing on real mobile devices in the real environment with real human interaction.
Daniel Knott loves digital products with high quality being web or native mobile applications. Currently, he is working as Head of Product Quality Engineering at MaibornWolff in Germany, where he shapes the future of testing for his clients. In the past 14 years, he worked as Lead Software Test Engineer for different native mobile apps and products.
Daniel wrote two books - Hands-On Mobile App Testing and Smartwatch App Testing and is a frequent blogger at www.adventuresinqa.com and conference speaker. Since 2022 he also created his YouTube Channel about Software Testing.
Get in touch with him via
- LinkedIn: Profile (https://www.linkedin.com/in/daniel-knott/)
- Twitter: @dnlkntt (https://twitter.com/dnlkntt)
- YouTube: https://www.youtube.com/c/DanielKnott/