Mobile and Responsive Design Testing: Solving the Combinatorics Problem

This is a guest post by Matthew Heusser.

It seems so simple. Click the first textbox, type four, click the second, type five, press submit, see “9” appear in the total field. Easy enough. Everything even works!

Of course, our real testing is more complex than that. We have the slider for the mortgage amount, the duration of the loan, the percentage down. Perhaps it has a login workflow, creating an account, or the checkout process.

Simple, repeatable, predictable. Perfect work for a computer to do — until you try to resize it. That’s when everything falls apart. At least, at certain resolutions.

Today’s websites tend to be “responsive design” and change themselves to fit the screen. Mobile applications can be responsive as well, scaling up to fit your iPad and down to fit your oldest-smallest-yet-still supported iPhone. And Android? Forget about it.

With mobile and responsive design, you write once, then test over and over again.

Modern Test Case Management Software for QA and Development Teams

The Combinatorial Problem

Two platforms: Android and Apple (2).

Three form factors: Phone, Tablet, Laptop. (2)

If browser-based, at least two browsers: FireFox, Google Chrome, and either Safari or Microsoft. (2)

You probably want to test the current browser version and the version before. (2)

You might want to test the current operating system and the version before. (2)

Plus, for each of the form factor/platform combinations, there are probably at least three popular devices (3).

2 x 2 x 2 x 2 x 2 x 3. Ninety-six test scenarios. Ain’t nobody got time for that.

What most teams do is the best they possibly can. Pick a few of the most popular devices, say, the ones the company owns plus the one in your pocket, do the basic tests, then explore a little, and call it a day. Those are the ones that even recognize the combinatorics problem at all. Plenty of organizations will just do the “click-click-click submit” test and call it a day.

We can do better.

Solving the Combinatorial Problem

The problem we are talking about is one of coverage. We might call it platform coverage. This is a problem that any good tester can identify.

The better the tester, the more hidden combinations we find. This is why the junior tester is done so much more quickly than the senior.

The difference is, the senior tester finds the bugs before they get to production.

All-Pairs, also called pairwise testing, is one approach to this problem. It reduces the number of test scenarios to all pairs of all the factors. Some tools allow the test designer to assign a weight. So, for example, when I was testing a luxury e-commerce website, we found that the money all flowed through Google Chrome and iPhone devices. It is possible to weight the tests so that when the number is not perfectly symmetrical, Chrome and iPhone come up more often.

But what do we do about automation?

Test Tooling and the Combinations

One option is to test everything. Test it all, in the cloud, all the time. To do that, you first need to write the test code, then manage the parallelism, the splitting up between 96 devices. Then either run on simulators (which you might not trust), or pay to rent all the devices. Even with all-pairs, you might get the total down to twenty. With a lot of research on who does what on which platform, you might reduce the total to ten or twelve. Even then, you have to find the bugs, which are almost certainly not going to be, “fail to add two numbers,” and more likely to be “that just looks wrong” or “this button is unclickable because something else is covering it.”

The two most common next steps are to either use visual tools, which notice pixel-level changes and allow quick authorizations (“yes, that change is expected, make this image the new expected one) and integrated responsive tools. Integrated responsive test automation tools are what they sound like — they integrate the user interface into components that scale up as the screen size scales up, making it possible to have just one test at multiple resolutions. Doing that usually requires that the developers use a responsive design tool that can scale the same way.

For today, keep in mind that there are plenty of failure modes when it comes to form factor. Consider them, and have a plan, or else they certainly will consider you.

All-in-one Test Automation Cross-Technology | Cross-Device | Cross-Platform

Matthew Heusser is the Managing Director of Excelon Development, with expertise in project management, development, writing, and systems improvement. And yes, he does software testing too.

In This Article:

Sign up for our newsletter

Share this article

Other Blogs

General, Agile, Software Quality

How to Identify, Fix, and Prevent Flaky Tests

In the dynamic world of software testing, flaky tests are like unwelcome ghosts in the machine—appearing and disappearing unpredictably and undermining the reliability of your testing suite.  Flaky tests are inconsistent—passing at times and failin...

Software Quality

Test Planning: A Comprehensive Guide for Success

A comprehensive test plan is the cornerstone of successful software testing, serving as a strategic document guiding the testing team throughout the Software Development Life Cycle (SDLC). A test plan document is a record of the test planning process that d...

Software Quality, Business

Managing Distributed QA Teams

In today’s landscape of work, organizations everywhere are not just accepting remote and hybrid teams—they’re fully embracing them. So what does that mean for your QA team? While QA lends itself well to a distributed work environment, there ar...