Sending your automated test results to TestRail

Sending your automated test results to TestRail

This article is part of a series of blog posts for Streamlining test automation with TestRail, which aim to guide you through centralizing your test automation efforts in TestRail, as per the diagram below.

ZYQ a2xlS7OMd8Y2PmHd 7vDQb9FTLyuT jTkr4 BabomnFg08tsbBgCNzXb0z8Y562JOxoUzw01dgceW M6dD cVB1

If you followed the previous articles in this series of blog posts, you should now have your test execution pipeline set up and your test results stored on Jenkins as artifacts. If you belong to an organization that values consistent test management and visibility, you will want to manage your automated test results within your test management tool so that you can centralize all your testing activities and generate consolidated and rich reports.

The TestRail CLI

To make this process as effortless and seamless as possible, TestRail provides a CLI to upload your test results, as long as your test automation framework can generate JUnit-style reports. The TestRail CLI parses JUnit reports, automatically creates test cases on TestRail, creates test runs, and uploads the results to the respective tests.

The TestRail CLI is a tool developed in Python and available in the Python Package Index (PyPI).  To make use of this tool, all you need is to have Python available on your system (which we already configured for the Pytest project). You can install it by using the command below.

$ pip install trcli

To enable using the TestRail CLI on your TestRail instance, you will only have to do some minor configurations.  See the TestRail CLI documentation for the required steps and to get acquainted with the features available through the CLI.

Configuring the build step to use the TestRail CLI

Let’s now jump back to our Jenkins job and quickly make the necessary changes to upload your test results to TestRail.

1. Edit the Execute shell step Command, adjusting it to upload the test results using the TestRail CLI.

Q2nFcESj7zxxdCdJhIRpQzZJQ4s2xbziYsjOQidV lPXdRQeKZzOsB CMYGm1U39xs06VovWY o0y5efQwfXU6kmcQEty6WOFji8Qee JL KGi3JE5lCCSOjnd87HLX3tDG6c vChiTkQGiXjdGHbZ4

Shell script breakdown

Below is a copy of the script used above and a brief explanation of each section.

# Setup virtual environment
python3.10 -m venv ./venv
. ./venv/bin/activate

# Install and run test project
pip install -r requirements.txt
pytest --junitxml "reports/junit-report.xml" "./tests" || pytest_exit_code=$?

# Install TestRail CLI and upload test results
pip install trcli
trcli -y 
  -h "https://INSTANCE.testrail.io" 
  -u "USER_EMAIL" 
  -p "API_TOKEN" 
  --project "PROJECT NAME" 
  parse_junit 
  --title "Automated Test Run" 
  --run-description "CI Build: ${BUILD_URL}" 
  -f "reports/junit-report.xml"
    
# Exit with Pytest exit code
exit $pytest_exit_code

# Install and run test project

The first change is on the pytest command. Notice the --junitxml "reports/junit-report.xml" option, which commands Pytest to generate a JUnit style report to be used by the TestRail CLI. You can find more details about this on the documentation relating to Integrating with Pytest. Another detail is the || pytest_exit_code=$? after the pytest command. This tells the shell to proceed even if tests fail, which causes Pytest to return a failure exit code and would otherwise stop the whole job execution without uploading the test results to TestRail. We will later exit the script with the pytest_exit_code to either pass or fail the build according to the test results. There are multiple ways to achieve this behavior and plugins that may help you doing so, this was the simplest way to fulfill the purpose of this example.

# Install TestRail CLI and upload test results

If the test execution finished successfully (whether there were failing tests or not), you should now have a JUnit style report to be able to upload your results to TestRail. To do so, you just install the TestRail CLI using pip and then execute it with your TestRail instance details. You’ll need to replace the placeholders for the -h, -u, -p and --project. Another small but useful detail on the trcli call in this script is the option --run-description "CI Build: ${BUILD_URL}". This will add the current Jenkins build url to your TestRail test run, so you can easily navigate back to it. As a final recommendation, do not use your password or API token directly on the script, make use of the Jenkins credentials manager.

# Exit with Pytest exit code

As mentioned before, we simply exit the script with the original Pytest exit code, which will inform Jenkins of the result of the actual automated tests execution.

Running the job

1. Now that we’ve completed changing the build step, we can run the job again. To do so, you simply hit the Build Now action on the left side panel.

5_run_job.PNG

2. After the job is complete, you should be able to see that the script was executed as expected by peeking at the logs on the Console Output page of the build. If the test execution completed without any bumps and the TestRail CLI was properly configured, you should be able to see that it submitted the test results to your TestRail instance on the build logs.

3. Finally, if you open your TestRail instance and go to the Test Runs & Results page, you should be able to find the newly created execution named Automated Test Run and see your test results, along with the link to your Jenkins job build that generated this test run. Notice the run description link so you can navigate to Jenkins and further debug your test results.

IazTLh7kwTXY2074 7th42LU lAW 8tLOATBN CzQd lDnFQ7isP85vHR9rSp ipBvB9m0OD7L17okKcBWfftyUMeGNj9ryyCkIORoOaqACzhZMt TfIvL WuYIsdWZcXM5R UadB26gmTRqE59TwVk

Final result

After implementing all the instructions in this article, you will have completed steps 3, 4, and 5 in the diagram below.

VVX92MxhBMEo 68bEY5XixvaBtsIt7SLF3h3y3ZrrFCXK7liJ3L2gYEjpf5zPzqTBKcTAU8fLm1THHCokTsZhzwYds mtNGsLXdlPZ6gtlFiLRC YgL2F1d v6u5v3SXqerlOA7Vket6Cm7BMb9fQw

Now that you have centralized your test results in TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results.

At this point, you still have to go into Jenkins to trigger your automated tests job before you have your test results on TestRail. But what if you could trigger the job directly from TestRail? To learn how to achieve this, please see the article referring to the 4th and last step in this series of blog posts – Triggering the Jenkins build from TestRail using UI Scripts.

In This Article:

Sign up for our newsletter

Share this article

Other Blogs

Automation

Test Automation Strategy Guide: Best Practices & Checklist

In the race to meet escalating customer demands and outpace competition by swiftly deploying software and new features, treating automated testing as a luxury is a risk you can’t afford. While test automation holds the potential to expedite your time ...

Automation, Programming

How to Report On Test Automation (Tools + Metrics)

Test automation streamlines repetitive tasks, accelerates testing processes, and ensures consistent, reliable results in software testing. Reporting on test automation provides actionable insights and visibility into the test outcomes, enabling teams to mak...

Agile, Integrations, Jira

Jira for Test Management: Options, Challenges, and Solutions

Learn about the pros/cons of using Jira as a test management solution, alternatives, & how teams that use Jira manage their testing.