Orchestrating an automated tests project with Jenkins as your CI tool

TestRail New Website Graphics NG V2 Blog 9

This article is part of a series of blog posts for Streamlining test automation with TestRail, which aim to guide you through centralizing your test automation efforts in TestRail, as per the diagram below.


In this example, we will use Jenkins as a CI tool to fetch and execute the automated tests project. In this article, we’ll explain how to create and configure a Jenkins job to fetch and execute the automated tests project you have been working on, on your local machine so far. In the end, you will achieve the same result you previously had (executing steps 2 and 3 in the image below), except it will be integrated in your CI tool.

Setting up a Jenkins instance

Note: If you already have a Jenkins instance running and an agent running Python 3.10.x, you can skip this section.

In case you want to quickly setup a Jenkins instance to follow the next steps, you can build this custom Jenkins Dockerfile (based on the official Jenkins Docker image) below and run it by following these instructions:

1. Open your command line on the location you saved the file to

2. Execute the commands below

docker build -t jenkins:local
docker run -p 8080:8080 -p 50000:50000 jenkins:local

3. Copy the key on the console and open http://localhost:8080 on your browser

4. Follow the install wizard (you can proceed using all the defaults)

Creating a job in Jenkins

Setting up a job in Jenkins is very straightforward. Just follow these steps:

1. Open your Jenkins instance and click New Item


2. Enter your New Item name, select Freestyle project and click OK


Setting up the Git repository

When creating the job, you can configure the Source Code Management tool you are using. The first action when you trigger the job will be fetching the code. To do so, follow these steps:

1. On the Source Code Management section, start by selecting Git

2. Set the Repository URL (in our example, https://github.com/gurock/automation-frameworks-integration.git)

3. Set the Branch Specifier (in our example, */main)


Configuring the build step

The build steps are where you define what you intend your job to execute. In this case we want to execute our automated tests project and we will use a shell to do so.

1. On the Build section, use the Add build step dropdown to add an Execute shell step


2. On the Execute shell step Command, insert the script with the commands required to setup the environment and execute the tests project

6z XNMQyHVIPotIbrLKWBQm99t4Wv6J8ViodGXneZPbA41VGwUA0jECHxgHLBQ7u 2HH91ZI APp2 W1u4KHx BbyyOSXEivLrng7EM TKk XY8WG8Hs5Vhlo 9IzmAlKIKUHT o9m7cIWqOikYd5So

Shell script breakdown

Below is a copy of the script used above and a brief explanation of each section.

# Go to test project folder
cd samples/python/pytest

# Setup virtual environment
python3.10 -m venv ./venv
. ./venv/bin/activate

# Install and run test project
pip install -r requirements.txt
pytest "./tests"

# Go to test project folder

The script starts by navigating to the Pytest sample project base folder which contains the automated tests project we want to execute.

# Setup virtual environment

The script starts by creating a Python virtual environment dedicated to this project, which is a good practice to make sure you start from a consistent baseline. After the virtual environment is created, you need to activate it, which means your shell will use it instead of the global Python installation.

# Install and run test project

After having activated the virtual environment, it simply installs the Pytest tests project and executes it.

Running the job

1. After you have saved the job with the configurations above, it’s time to run it. To do so, you simply hit the Build Now action on the left side panel.


2. After the job is complete you should be able to see the script was executed as expected by peeking at the logs on the Console Output page of the build.

FRe7xZg G5zUmgpMI0zDLFw43LcG b2G8Km2YBV3978hvaa1qJAoTYDcYN2gAiL6j6 gnEphpX7g6VIR65Aw7HlNGvCJ2TkAiD0l5Ld9Ww4VjTOh54PTs

Final result

You have just automated the execution of your automated tests project. If this instance of Jenkins is online and accessible to your coworkers, they can now also execute the automated tests suite with the click of a button. Besides this, there are other advantages, such as the tests always running in a controlled and consistent environment, which makes the execution replicable and reduces instability due to environmental changes. Below is a diagram with the pieces you’ve put together so far, representing a fully working automated tests project within a CI tool.


To bring even more visibility and the ability to further analyze and report on your automated test results, you can centralize these results on TestRail, consolidating your manual and automated testing efforts. To learn how to achieve this, please see the article referring to step 3 on this series of blog posts – Sending automated test results to TestRail.

How useful was this post?

Click on a star to rate it!

Stay ahead in Test Management.

Follow us on social media!

Help us improve this page!

What problem are you trying to solve?

In This Article:

There are no headings in this document.

Sign up for our newsletter

Other Blogs

Agile, Integrations, Jira

How Software Development Company ELEKS Increased Visibility and Productivity by Integrating Jira with TestRail

Read more about how the QA team at ELEKS evaluated different test management solutions and why they ultimately chose TestRail.

Agile, Integrations, Jira

How to Report on Traceability and Test Coverage in Jira

This post covers how to show test coverage, manage requirements traceability, and create a traceability report in Jira.

Agile, Integrations, Jira

Jira for Test Management: Options, Challenges, and Solutions

Learn about the pros/cons of using Jira as a test management solution, alternatives, & how teams that use Jira manage their testing. 

Build quality processes and release with confidence