Testing Push Notifications Demystified: An Engineer's FAQ

Push notifications are a critical feature within mobile apps, with users counting on them for reminders and alerts, and app publishers relying on them to keep users engaged and bring them back into the app. Users rely on these reminders and messages for timely notice of events like news, market trends, and group messages. As a result, it's important to test your push notifications with each code release. If they don't work, your users will miss out on critical information and you risk losing them to competitors. 

But, testing push notifications isn't easy. They present several challenges that many other features don't share. They behave differently on Android and iOS, and users have significant control over how they act on their devices. We've assembled a list of frequently asked questions (FAQs) and answers about testing mobile push notifications below. 

Let's dive in! 

Why Test Push Notifications?

While they seem straightforward, push notifications have a lot of moving parts. For example, users should receive your messages even when your application is closed. They should also work when a user's phone is locked, and if they contain clickable content, the link needs to be valid and point to the right location inside your application.

So, make complete verification of your push notifications a part of your test suite.

What Should I Test With Push Notifications?

To test mobile push notifications, you need to cover the different circumstances under which your users might receive them. This includes different configurations, various application states, diverse user options, and as many different hardware and operating system versions are you can cover. 

For example: 

• Are they visible when your application is running?
• Does the device still receive and display them after a user has closed your app?
• What happens when the device is in "do not disturb" mode?
• Are the notifications visible when another application is running?
• Does setting the device for a different timezone prevent delivery or change how the alert works?
• Is the alert clickable? Does the content display correctly when it's clicked?
• Do the notifications display the correct options? (Sounds, type, icons, etc.)

This isn't an exhaustive list of features, but it should give you an idea of where to start with your test cases for push notifications. 

What Tools Are Used for Push Notification Testing?

As of XCode 11.4 beta, you can test push notifications for iOS with Apple's iOS Simulator. You can register for them in your application, drag-and-drop APN files to the simulator, and simulate them from the command line. You can also test push notifications in the Android emulator. Like the iOS Simulator, it can register for notifications and display them. 

Of course, the best test is with a real device. The emulators make it easier to test with a variety of screen sizes, operating system versions, and configurations. Still, they don't account for subtleties like how applications act when they are running in the background or how they interact with other processes on native hardware. The final test of a new release needs to be on real hardware and has to include push notifications. 

In addition to simulators and real hardware, you can perform your end-to-end testing with the notification service you use in production. For example, Firebase has instructions for testing Android and iOS applications here. OneSignal has an FAQ here. 

‍

What Are the Major Challenges of Push Notification Testing?

While push notifications seem to be simply a matter of sending an asynchronous message to your application, there's a lot that can go wrong. 

As discussed in the previous question, complete testing of push notifications requires testing with real devices. (This is true of any mobile app, not just an app with push notifications.) Assembling a set of mobile devices with the right mix of versions and configurations is hard but not impossible. 

• Do you use more than one messaging provider? How do their APIs and message formats differ? Do they deliver messages to client devices with the same semantics?
• Are you accounting for the many different device configurations your users are using? How do features like scheduled delivery and do not disturb affect your application?
• Different screen sizes, operating system versions, and configurations will change how your messages behave.
• How will new updates to mobile operating systems and hardware affect your push notifications?
• As we covered above, testing with real devices is a requirement for complete coverage.

What Are the Differences Between iOS and Android Push Notification Testing?

The obvious difference between testing iOS and Android push notifications is their GUIs. You'll need to write different tests to verify the contents of your notifications. But the differences don't stop there. 

iOS and Android manage notification delivery in very different ways. Here are a few things to take into account when testing them: 

• Apple users have to opt-in to notifications from your application. So, when you're setting up integration tests that cover installing a new copy of your application, your tests need to allow for enabling messages.
• iOS users also have control over how and when they are displayed. They can disable notifications with Do Not Disturb and defer their delivery to later in the day.
• Android and iOS manage notifications on the device lock screen differently, too. You'll need to code automated tests for the different ways users view and delete messages.

What Are Some Common Mistakes in Push Notification Testing?

You can improve your application's stability and customer satisfaction by avoiding a few common testing errors: 

‍

• The most common mistake in testing push notifications is not testing them at all. If this FAQ has a theme, it's that even though notifications are primarily tested "out of app" and seem simple on the surface, they require extra care and attention.
• Failing to test links to rich content and applications. Not testing these links can lead to a poor experience and even application crashes.
• Relying on simulators and emulators for testing. They can handle push notifications, even on iOS now, but testing on them is insufficient. Notifications need to be to allow for applications running in the background and contending for system resources.

How Can I Integrate Push Notification Testing into My CI/CD Pipelines?

While push notifications require extra care, that doesn't mean you can't include them in your CI/CD pipelines. You can integrate the tests into automated builds and deploys and adjust the nature of the tests depending on the type of build. For example, real device tests are more expensive in terms of both time and money, and adding them to every build would slow down development and may even lead to developers skipping the test altogether. 

Here's a possible approach to testing push notifications in a CI/CD pipeline: 

• Development builds test notifications in a simulator. To speed the process, the test generates messages locally using simctl for iOS or the loopback API for Android.
• User acceptance testing also uses a simulator. But it sends the notifications via your messaging service as an extra verification step.
• The pipelines deliver release builds to real devices. The apps receive notifications via the production messaging services.

Test Your Push Notifications

Push notifications are an important feature, and supporting them can get complicated. But with proper care and careful testing, you can build and release an application with reliable and useful messages for your users.

This post was written by Eric Goebelbecker. Eric has worked in the financial markets in New York City for 25 years, developing infrastructure for market data and financial information exchange (FIX) protocol networks. He loves to talk about what makes teams effective (or not so effective!).