This is a guest post by Matthew Heusser.
Building on our previous jUnit integration article, we’ll make the tests execute as part of the CI/CD (continuous integration and continuous delivery) pipeline using examples in Java. We’ll explain the “why,” and the “what” with an example in Gitlab.
Why Integrate Test Management and CI/CD?
Continuous integration (CI) has many more benefits than “automated builds” or even “automated running of (perhaps ‘some’) tests.” By performing a build for every change in version control, CI can track the exact change that breaks the build or causes a test to fail. This means tracing a failure back to the lines of code that created the error, the person who made the change, the time, and likely the feature, story, or piece of business functionality that change was designed to enable. Brett Schuchert, a leading consultant at Industrial Logic known for his work on the Framework for Integrated Tests (Fitnesse), suggests that small commits make the difference here. Our experience is that by tracking a single change at a time, the debug and fix time is reduced between 25-90%. Presented with those numbers, Schuchert replied “I think the actual value/multiplier is much higher than your suggestion because having this information might make otherwise unsolvable issues solvable.” These small changes also enable continuous delivery, the “CD” in CI/CD.
To find errors beyond simple compile problems, continuous integration (CI) systems need to run some amount of automated tests. That might be unit tests, it might be API tests, it might be front-end javascript tests or even end-to-end tests with a tool like Selenium. A few companies, particularly software startups, run only these automated tests. Most companies with any legacy software, however, have a hybrid that combines some tool results and also some human exploration.
Now think about the quality of the system in flight. Where can managers or executives go to get information on the next release of the software, the changes, and the latest build? CI/CD results are highly technical and limited to what the tooling can do automatically. Test case management systems like TestRail can provide a single view of results, but only if those results are unified. That means humans updating test cases for a test run, and something updating the test case management system on a CI/CD run. Our suggestion today is that you let the CI system update the test case management, using the APIs published by the test case management system.
Here’s what implementing that change looks like in CI/CD.
Integrating Test Case Management and the CI/CD Pipeline
It all starts with a single change. In our example which tests the triangle problem, the code could be changed to take in a larger integer type. This will cause the tests that check for boundary errors to fail, as the boundaries have now changed to larger numbers. The programmer might run the unit tests associated with the object they are testing but might not test the entire suite, or the higher-level tests. Often the change could either introduce new expectations, making other tests “fail” as the software no longer meets those expectations, or simply break other pieces of functionality. For example, an API change that fundamentally changes the structure of the API result could cause a front-end javascript problem. The illustration below shows the full scope of a CI/CD pipeline.
Source: Learn.Gitlab.com
The smaller rectangle – the CI pipeline – is our focus today. This is where the test failures (or successes) will happen. A CI pipeline that runs will get a “green build” and perhaps build a package. A continuous delivery pipeline that succeeds might create a ticket for someone to do any final checks — after the checks pass, the decision-maker could “push” code to production in one step. A continuous deployment pipeline will push to production on success. On failure, all systems will turn, stop, and send notifications to the offenders and related managers.
Those successes and failures come back in two or three ways. They can be a text file with results, an exit code from the test application, or, perhaps, text sent to STDOUT that can be redirected or captured. The CI pipeline needs to get those results in order to light up “RED” (failing) or “GREEN” (everything passes). At its heart, you can think of a CI tool as an extremely fancy collection of command-line runs combined, if-then statements, configuration, and dashboarding of results. Point the CI tool at a codebase, tell it how to perform the build and interpret results, and look at the graphics. While that is overly simplistic, it gives enough of a view to understanding how to update test management: write a little program to create a new test run, then, for each test case, update the test case for that test run as pass or failed. Track the results and close the test run as passed or failed.
Here’s how to do it.
How to integrate TestRail and GitLab
This example will start with a real project in GitHub, with Git installed locally and a GitLab runner on a local machine that will work as a worker to build the software. This tutorial will provide you with the exact steps to follow and create your own pipeline.
First, create a Github account and fork – the JavaSamples repository is a good example. When the author did this he got a new URL https://github.com/heusserm/hellogitworld. Now click code on the dropdown, copy on the local machine, go to your code directory, and from the command line run replace “heusserm” with your username:
Git clone [email protected]:heusserm/JavaSamples.git
Now in GitLab, from the home screen, select New Project, Run CI/CD for an external repository, and either click “Connect repositories from GitHub” or else paste your Git repository URL.
Next install a runner for GitLab (installer for Windows, Linux, Mac OS). The runner needs to be registered. Get the registration code in GitLab from Project Settings->CI/CD->Runners. This connects the runner to GitLab. Running from the local machine is the most convenient, and least secure/stable way to do this, but the environment will be perfectly set up so that compiles run properly.
Here’s a screen capture of the registration process:
Once the runner is registered, go back into Project Settings->CI/CD->Runners and check “run untagged jobs” then save changes. That makes this runner the default runner for all jobs.
Prior versions of the software had the TestRail user and password in the source code. As this is public in Github, we changed the code to pull usernames and passwords from the TestRail_USERNAME and TestRail_PASSWORD environmental variables. Otherwise, the code is the same as the JUnit/TestRail integration article, which includes a Java example of calling the TestRail API. The configuration file that drives Gitlab is called “.gitlab-ci.yml.” To make this work, we broke build, test, and report into different commands. Also, we added code to the report program to send an error code to the operating system if a test fails. This will tell GitLab something went wrong and report the failure.
Here’s the gitlab-ci.yml file, the file that tells GitLab how to perform build, test, and report.
stages:
– buildtestreport
build-job:
stage: buildtestreport
script:
– cd TestRail_triangle/Triangle_02/
– ./01build.sh
– ./02test.sh
– ./03report.sh
On the Mac, this author needed to upgrade the versions of brew, git-lfs, and git to get runners to work on GitLab through the firewall. Once that was done then every time a commit happened on JavaSamples, the runner kicked off and the CI pipeline ran. A typical GitLab pipeline would be very similar to this one, only separating the jobs and passing files as artifacts.
Once that was done, editing a Java file and pushing a change to GitLab kicked off a new pipeline run:
Click on the specific pipeline to drill into all the text output for the run.
And the change to the pipeline updated TestRail as well.
Common integration scenarios
The example above mapped test cases in TestRail to Test Cases in the test framework. Every time the software ran, it created a “test run” and closed it down. That is not the only way to do things. Another approach would be to create a single test case per test suite run – perhaps one for API and one for end-to-end tests. The third common category is unit tests; putting unit tests as test cases is not a common practice. With TestRail and CI/CD, though, it is relatively easy to have a single test case for each major group of unit tests, to create visibility of unit test results.
Some companies have a more complex CI/CD pipeline, that runs all the tests for every change, then merges those changes onto a tip or primary branch for release candidates. You might configure the pipeline to only create a test run for these “release candidate” builds. Likewise, it might make sense to create a test run and not close it, as the people doing other elements of testing will need to update their human test cases.
Another approach is to have one open test run and store it in CI in some sort of configuration file. The CI does not open a new run but instead updates the test cases on the existing run. When the deployment happens the deploying software can open a new test run and update the configuration file.
The “right” choice here will depend on what is right for your group. Having API access to actions in TestRail gives the computer the same capabilities as a human – the computer can integrate the work, which can lead to a single, integrated view of software quality. Executives have been asking for that for decades.
Why not give it to them? Check out our Integrating TestRail and JUnit article for our example of a computer program that built the software, ran the tests, then automatically created a test run and updated the test cases in TestRail.
