For a welcome change from testing the upcoming SmartInspect release, I decided to write some notes about how we test our software on multiple platforms.
Let’s begin with the simpler part, the SmartInspect Console. Since the tests for the Console are currently not automated, we maintain a large test plan which needs to be executed manually. We process this plan on every supported platform before we release a new version. Testing the Console on one platform basically means to grab a fresh copy of a Virtual Machine for this platform, install the current SmartInspect build on it and process the entire test plan. It’s simple but quite tedious. It normally takes an entire day or sometimes more to process one platform (including different platform versions).
For the libraries, we developed lots of unit tests for each library which test most of their functionality. Executing these tests takes only a few minutes. However, we faced a problem when we thought about running them on multiple platforms. Since our test frameworks originally needed a compiler to run, we would have needed to install a compiler for every language on every supported platform. This would have been highly impracticable. So we decided to adjust our build system to create nice precompiled packages for our tests. Testing a library for a specific platform basically means now to copy these packages to a new Virtual Machine, unzip them and start the automated tests.
We also thought about automating the testing of the libraries even further. The new VMware version contains a tool which can be used to start Virtual Machines directly from the command line. Combined with a few scripts, this could be used to automate the testing of the libraries on all supported platforms completely.
