This is a guest post by Matthew Heusser.
It seems so simple. Click the first textbox, type four, click the second, type five, press submit, see “9” appear in the total field. Easy enough. Everything even works!
Of course, our real testing is more complex than that. We have the slider for the mortgage amount, the duration of the loan, the percentage down. Perhaps it has a login workflow, creating an account, or the checkout process.
Simple, repeatable, predictable. Perfect work for a computer to do — until you try to resize it. That’s when everything falls apart. At least, at certain resolutions.
Today’s websites tend to be “responsive design” and change themselves to fit the screen. Mobile applications can be responsive as well, scaling up to fit your iPad and down to fit your oldest-smallest-yet-still supported iPhone. And Android? Forget about it.
With mobile and responsive design, you write once, then test over and over again.
The Combinatorial Problem
Two platforms: Android and Apple (2).
Three form factors: Phone, Tablet, Laptop. (2)
If browser-based, at least two browsers: FireFox, Google Chrome, and either Safari or Microsoft. (2)
You probably want to test the current browser version and the version before. (2)
You might want to test the current operating system and the version before. (2)
Plus, for each of the form factor/platform combinations, there are probably at least three popular devices (3).
2 x 2 x 2 x 2 x 2 x 3. Ninety-six test scenarios. Ain’t nobody got time for that.
What most teams do is the best they possibly can. Pick a few of the most popular devices, say, the ones the company owns plus the one in your pocket, do the basic tests, then explore a little, and call it a day. Those are the ones that even recognize the combinatorics problem at all. Plenty of organizations will just do the “click-click-click submit” test and call it a day.
We can do better.
Solving the Combinatorial Problem
The problem we are talking about is one of coverage. We might call it platform coverage. This is a problem that any good tester can identify.
The better the tester, the more hidden combinations we find. This is why the junior tester is done so much more quickly than the senior.
The difference is, the senior tester finds the bugs before they get to production.
All-Pairs, also called pairwise testing, is one approach to this problem. It reduces the number of test scenarios to all pairs of all the factors. Some tools allow the test designer to assign a weight. So, for example, when I was testing a luxury e-commerce website, we found that the money all flowed through Google Chrome and iPhone devices. It is possible to weight the tests so that when the number is not perfectly symmetrical, Chrome and iPhone come up more often.
But what do we do about automation?
Test Tooling and the Combinations
One option is to test everything. Test it all, in the cloud, all the time. To do that, you first need to write the test code, then manage the parallelism, the splitting up between 96 devices. Then either run on simulators (which you might not trust), or pay to rent all the devices. Even with all-pairs, you might get the total down to twenty. With a lot of research on who does what on which platform, you might reduce the total to ten or twelve. Even then, you have to find the bugs, which are almost certainly not going to be, “fail to add two numbers,” and more likely to be “that just looks wrong” or “this button is unclickable because something else is covering it.”
The two most common next steps are to either use visual tools, which notice pixel-level changes and allow quick authorizations (“yes, that change is expected, make this image the new expected one) and integrated responsive tools. Integrated responsive test automation tools are what they sound like — they integrate the user interface into components that scale up as the screen size scales up, making it possible to have just one test at multiple resolutions. Doing that usually requires that the developers use a responsive design tool that can scale the same way.
For today, keep in mind that there are plenty of failure modes when it comes to form factor. Consider them, and have a plan, or else they certainly will consider you.
Matthew Heusser is the Managing Director of Excelon Development, with expertise in project management, development, writing, and systems improvement. And yes, he does software testing too.
