This is a guest post by Shreya Bose
Test coverage and traceability are two of the most important metrics to track when testing. This post will cover why test coverage and traceability are essential in Jira and how to show test coverage, manage requirements traceability, and create a traceability report in Jira.
Requirements traceability is a way to measure test coverage and usually refers to tracking a business requirement across different stages of the development lifecycle (i.e., requirement gathering, design, development, testing, and maintenance.)
Traceability is essential for risk mitigation because it allows you to identify an item as problematic, understand what changes caused it, and determine how to address it. Full traceability is crucial to compliance and risk mitigation efforts, as seen in this case study from Convercent.
In Jira, traceability means being able to track each user story, requirement, or issue to a test and any resulting defects. You should be able to see linked test cases and results on individual requirements and generate a full traceability report showing your Jira issues and all their linked test artifacts at once.
Test coverage is a QA metric that measures how much of the product or application being tested is covered by tests. Test coverage also helps ensure that you have prepared, designed, and planned tests to cover every part of the application.
In Jira, test coverage helps us determine the number of tests run, passed, failed, blocked, etc. for every feature or product requirement we have written in Jira as a user story, epic, or issue. Typically, you can track Jira test coverage by linking user stories with their associated test cases and vice versa. Test coverage is vital to ensuring that no feature is going untested.
The idea is to know the percentage of requirements that have a test case written for them. The higher the percentage, the more confidence we can have that our testing activities will uncover any potential risks in our application.
Tracking test coverage status throughout the software development life cycle (SDLC) allows you to identify areas where you don’t have adequate coverage and take action to avoid any decrease in quality. This also helps to ensure that defects will be kept from production. If defects exist in the application, they will be caught during the testing phase rather than when the product is already live. This provides a brief overview of the current health of the development project.
Test coverage measures how much of the product being tested is covered by tests. Typically, a requirement is considered “covered” only if it has corresponding test cases against it and test engineers assigned to it.
Traceability refers to the established connections between various requirements levels, risk analysis, designs, and verification. A traceability matrix serves to map the test cases to the requirements. It serves as a checklist wherein all the requirements are listed, and the test cases covering the corresponding requirement are listed against each requirement.
But why should you be concerned? Here are the top seven reasons why tracking test coverage and maintaining traceability in Jira is important:
To create test coverage or traceability reports, you will first need to have requirements or user stories and test artifacts like test cases, tests, and defects.
Many teams that use Jira try writing test cases as subtasks in issues or creating custom issue types for testing. However, Jira is not designed for test management and doesn’t come with any kind of built-in test coverage or traceability reporting. In this blog post, we will use TestRail, a dedicated test management tool that integrates with Jira, to create test coverage and traceability reports against user stories tracked in Jira.
Most teams that track requirements or development tasks in Jira do so with issue types like epics or user stories. Your team may write detailed descriptions of the functions or capabilities the final product should have, or they might be defined as an informal, general explanation of a software feature written from the end user’s perspective.
The first step to creating a test coverage or traceability report is identifying which user stories have been prioritized for your next sprint or release and need to be tested.
When writing test cases for each requirement, there are a few options:
1. Write test cases as sub-tasks on an issue in Jira
2. Create a custom testing issue type or use a Jira add-on
3. Use a dedicated test case management tool
Ultimately, if you are trying to report on test coverage or traceability, the fastest way will be to use a test management tool with built-in coverage and traceability reporting.
As you are writing your test cases, make sure to link any requirements from your external tool. In TestRail, for example, you can copy/paste a Jira issue ID for your user stories or requirements into the References field on test cases or test results artifacts to indicate that the test is related and provide two-way visibility between the two platforms.
Creating a test coverage report will be a manual effort if you write your test cases as Jira issues or sub-tasks. In this scenario, the fastest way to create a coverage report would most likely be to copy the issue IDs for your user stories into a spreadsheet, then create a second column to write down the IDs for each of the test case issues you’ve created and linked to each user story, issue by issue. You could use spreadsheet functions to count up how many user stories IDs also have test case issue IDs and how many do not, and create a chart if you wanted.
For detailed steps to quickly create a coverage report in TestRail, check out TestRail’s test coverage report documentation or watch this video How To Set Up Test Coverage with Jira.
Once you have created enough test cases to adequately cover your full application, you can start running tests.
As you run your tests, it is important to link any defects you discover to ensure full traceability. In TestRail, when you add a test result in a run or plan, you can use the Defects field to link a defect or bug report from Jira or even create a new issue in Jira automatically by pushing a defect from TestRail.
Finally, once you have logged some test results or are trying to check whether a release is ready to deploy, you can generate a traceability report.
If you are creating a traceability report manually, you will most likely want to use a requirements traceability matrix (RTM) using an Excel spreadsheet or Google Sheets. A requirements traceability matrix can show the following:
Here are the steps to create a basic requirements traceability matrix. The steps are as follows:
1. Define your goal
Example: “I want to create a traceability matrix to know which tests and issues are impacted if a requirement changes.”
2. Gather all artifacts (i.e., Requirements, Test Cases, Test Results, Issues…etc.)
3. Create a template for the RTM by adding a column for each artifact
4. Add Requirements from the Requirement Document
Add Requirements descriptions and IDs
5. Add Test Cases from the Test Cases Document
Add Test cases that correspond to the requirements mentioned in the previous step
6. If applicable, add Test Results and Issues
Add the test results that correspond to the test cases mentioned in the previous step
7. Update the matrix when required
Update the matrix whenever there is a change in any artifacts to reflect the project’s current health.
Here is an example of a traceability matrix in a Google or Excel spreadsheet (Note: For various project iterations, there will usually be more columns).
To auto-generate a requirements traceability report in TestRail, start by opening a user story and clicking on the TestRail cases field to automatically open a new panel with the current test cases for this specific user story.
In Jira, click the button to add a test case and select the project you want your test case created in. From here, you will be redirected to the TestRail test case creation page with the user story issue ID already in place in the references field.
Next, your must name this first case. For this example, we will name this first test “successfully log in” and immediately save it by clicking “Add test case”.
The reference is now displayed as a link that you can hover over and edit. By clicking the reference, you’ll be redirected to Jira where you can open the TestRail cases panel and immediately see that your test case is now covering the user story through the reference on TestRail. Plus, if you click the test case link, you’ll navigate to the test case details in TestRail.
Note: The fields displayed can be customized when configuring the Jira integration on TestRail.
Now you can open the test cases list page and see your test case there. While on this page, you can quickly outline test scenarios and add the user story reference in bulk.
In this example, to add a couple of test cases to cover our login user story, we select “failed to login with invalid email,” “Failed to login with invalid password,” and “fail to log in without credentials.” In this example, we don’t have any associated references yet, but we can select all the test cases to quickly add the reference in bulk. To do this, just select the cases, click Edit, and then Edit selected.
Lastly, check the references field, add the reference to your login user story, and save the changes. You have now created traceability between these tests and our user story.
If your team uses Jira to manage software development, test coverage and traceability are two of the most important metrics to report on and track. Traceability helps to ensure visibility, manage continuous change, streamline the auditing process, and make test management more effective overall. Moreover, test coverage saves you time, boosts ROI, and promotes smoother testing cycles. Together traceability and test coverage help minimize risk and ensure that your QA testing process is comprehensive and secure.
Help us improve this page!
What problem are you trying to solve?
Read more about how the QA team at ELEKS evaluated different test management solutions and why they ultimately chose TestRail.
Learn about the pros/cons of using Jira as a test management solution, alternatives, & how teams that use Jira manage their testing.
a list of the top 5 QA metrics that provide valuable metric data that can improve the software application’s quality continuously.