Streamlining test automation with TestRail

Streamlining test automation with TestRail

TestRail defines itself as a test management tool that allows you to centralize all testing activities like test design, test planning, test execution, and reporting, all in one place. On top of that, TestRail has out-of-the-box integration with a wide range of issue trackers and other mechanisms you can use to create and customize integrations with other tools.

This article will explore how you can leverage TestRail by using UI Scripts and the TestRail API to fully streamline your automated tests execution and debugging from inside TestRail. For demonstration purposes, we will use Pytest as the test automation framework, GitHub for source code management, Jenkins as the test automation orchestrator, and the TestRail CLI as an interface to the TestRail API to upload automated test results. We will give a step-by-step overview of all the necessary blocks to reach the end goal, followed by a series of articles with more detailed instructions on achieving each building block’s end result.

Step 1: Setting up a Pytest automated tests project

We’ll start by exploring a Pytest automated tests sample project, installing it, and executing the tests contained in it in your local machine. This project is hosted on TestRail’s Automation Frameworks Integration GitHub repository, so we will be using GitHub as our Source Code Management platform, from where you can fetch the project source code on demand. You should be able to quickly achieve the workflow on the image below by just installing Python on your system and executing a couple of commands to install and execute the project on your terminal. After completing this set up, you can execute the automated tests project as many times as you wish on your machine and customize it to your needs.

3tLJfegzCTNJYa8SiNxfUrtGRsA24D48OPOo6C3QSE32Dtghl41wlBLdpWzGgiAcKbzquXtYHG6wB nKskC pNXcILBweGvO0WhzS4dpZw7UX5gKfruwDq7l4BwqPy37O291e 6GVlpcz77GYDzH0A

Step 2: Orchestrating an automated tests project with Jenkins as your CI tool

Executing automated tests on your local machine can be enough when you’re just starting, but it is not scalable and has downsides such as creating team dependencies on one person, being unable to reproduce the same project execution environment, or simply lacking visibility foryour organization. To avoid these downfalls, you should automate the orchestration of your automated tests project using a CI tool, in this case, Jenkins. We will create a job that fetches the source code from GitHub and executes the tests in it using a shell script, as you would on your local machine, ending up with a workflow similar to the first one but on a proper CI system.

i80yMAt9erW8s4c6aQ60hjFv N U77OgwDFMp34Sc03Vf5M9ppfuuHGwzAjd6db5Xrfn9VbrzOZzcR7M7Br0nKNdQF5e92YNDEoUkw9J 6suJ4jEwbHYCYC2Req8WwB73BmNXhQ7NfsblFQl1NDAHg

Step 3: Sending automated test results to TestRail

Now that you have your tests successfully running on your CI tool, it’s time to bring your test results to light by integrating them into TestRail. To do that, we’ll leverage the TestRail CLI by using its ability to parse JUnit reports and upload their results to TestRail. First, you’ll need to configure Pytest to generate a JUnit style report, which is supported out-of-the-box and achieved through a simple command line argument. Now that Pytest is configured to generate the JUnit report, you just need to install the TestRail CLI and you can send your test results to TestRail with one simple command. You will then have achieved the workflow on the image below.

VVX92MxhBMEo 68bEY5XixvaBtsIt7SLF3h3y3ZrrFCXK7liJ3L2gYEjpf5zPzqTBKcTAU8fLm1THHCokTsZhzwYds mtNGsLXdlPZ6gtlFiLRC YgL2F1d v6u5v3SXqerlOA7Vket6Cm7BMb9fQw

As a result of this workflow, each time the Jenkins job runs, the TestRail CLI creates a new test run with your test results on TestRail. You can see these results in your TestRail’s Test Runs & Results page and act on them as you would on any other test run, dig into test results to debug failing tests, change test results, add comments, etc.

UZ SaAiVz1mo8oWDeowjSGA7jnCmUg2Fz5bFCuqd3m S bZZ94f4HVaZko6H2nD WvGXUFbr4o3thJbrqDDkadcp6HU2Al1OeN4osH0WuiPWxTxSW f1OWzI0dKSd62NT 39NVYn8dvPYlHupfh35Q

Step 4: Triggering the Jenkins job from TestRail using UI Scripts

Lastly, we’ll use UI Scripts to close the circle by adding a widget on TestRail to trigger your Jenkins job and retrieve build information to quickly understand what has been going on with your latest builds. We provide a Jenkins Builds UI Script that is hosted on our TestRail Customizations GitHub repository. You simply change the configuration variables at the beginning of the script according to your needs and add it through the Administration > Customizations > UI Scripts page on your TestRail instance.

0ZkVA2WxtjUK7JDjLouVoOb9yrqByfFMlF 5KvU0mQR1KoqmLjyrsFk90Qs PIwaNr YAUXIlTYXeDktSW3bjX5HqMhslEVV 80jBZYLvbFerbuP0Z s GIuKYwojR JA0QEC7VzgKo2xSijs Lbk

In TestRail, you will see that if you navigate to the Test Runs & Results page, there’s a new widget on the right-hand panel with a trigger button for the configured job and the latest builds. This widget allows you to trigger the job, check the latest builds status through the color code, and navigate to the builds by clicking on the respective links.

zkDdLvLuMvMcTV55kD96kJQ OsFbA6wxedqZZZoO4MlLB82vfqhbxK3spJvhrvn2js80SaeVs3OasIQgFVmsA63FqJ2bXvSp19b r OvTijyPSe00PKGN8BfyYqm8qcnhq2tfdZ9bWGwBlHhW434XQ

Final takeaways

With this workflow in place, triggering and accessing automated tests information becomes part of TestRail. Any user who is performing his daily activities in TestRail can start an automated tests suite, and receive the test results as a test run as soon as it is finished on the CI tool, without ever needing to navigate outside TestRail.

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing information on reports. By doing so, you can get solid insights on your app’s test coverage and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would for your manual test results.

For more detailed instructions on how to implement this workflow, please see the next articles in this blog series:

  1. Setting up a Pytest automated tests project
  2. Orchestrating an automated tests project with Jenkins as your CI tool
  3. Sending automated test results to TestRail
  4. Triggering the Jenkins job from TestRail using UI Scripts

Documentation references:

In This Article:

Sign up for our newsletter

Share this article

Other Blogs

Automation, Programming

How to Report On Test Automation (Tools + Metrics)

Test automation streamlines repetitive tasks, accelerates testing processes, and ensures consistent, reliable results in software testing. Reporting on test automation provides actionable insights and visibility into the test outcomes, enabling teams to mak...

Agile, Integrations, Jira

Jira for Test Management: Options, Challenges, and Solutions

Learn about the pros/cons of using Jira as a test management solution, alternatives, & how teams that use Jira manage their testing. 

Software Quality, Integrations, TestRail

How a Document Management Company Streamlined Testing and Boosted Efficiency with TestRail and Reflect

A leading document management company in the document management industry has made significant strides in streamlining digital content access and organization with its cloud-based document management platform. Their suite of software products enables compan...