One common question we receive from new TestRail users is: “How do I trace requirements using TestRail?” Sometimes, folks even specifically ask, “Does TestRail support RTM (Requirements Traceability Matrix) to trace the test coverage?”
In this post, you will learn:
As per Wikipedia, traceability is the capability to trace something. In some cases, it is interpreted as the ability to verify the history, location, or application of an item by means of documented recorded identification.
In IT and Software Engineering, traceability usually refers to the ability to track a business requirement across different stages of the development lifecycle like requirement gathering, design, development, testing, and maintenance.
Traceability works by linking different artifacts like requirements/user stories/epics to their corresponding test cases, test runs, test execution results (including defect details if any) and vice versa.
Requirements traceability ensures that requirements are met and we have built and tested the product correctly.
Increasing test coverage reduces defect leakages / QA misses (before the product is released to the world), and a RTM establishes a way to verify that we (as a testing team) have adequate coverage built into our test planning and execution.
Traceability also helps create a snapshot to identify coverage gaps and gives us visibility throughout the development lifecycle, thus accelerating the development and reducing waste of resources like time and effort. In this capacity, we can start to track coverage as a metric that helps us determine the number of tests Run, Passed, Failed, or Blocked, etc., for every requirement, thus providing a brief overview of the current health of the development project.
Moreover, traceability also plays an essential role in analyzing the impact when any related artifacts (like requirements, test cases) change. For example, if our requirement with the ID R01 changes, its related test cases T01 and T02 need to be modified to keep the document consistent.
Ultimately, the goal for traceability is to help you plan and manage testing activities (including defect management) better.
Usually, when someone refers to “RTM” in context of software testing, they are referring to a worksheet—usually produced in an Excel workbook or Google Sheets—that contains the requirements for their project with all its possible test scenarios and cases and their current state (i.e., if they have been passed or failed).
Let’s consider a scenario where we have the following business requirement like “User should be able to sign up using email and Facebook” with 2 test cases that are part of Sprint 1, namely:
1. T01: Sign up using Email
2. T02: Sign up using Facebook
As an example, let’s say that when these test cases were executed, the results were Passed and Failed respectively. In this case, the basic traceability document might look like this :
From this snapshot, we can gather a lot of information at a glance, like:
It’s quite simple to create a basic RTM document. The steps are as follows :
Step 1: Define your goal (To make sure you’re gathering the right information for your traceability matrix)
Ex: I want to create a traceability matrix so that I know which tests and issues are impacted if a requirement changes.
Step 2: Gather all the artifacts (Like Requirements, Test Cases, Test Results, Issues / Bugs)
Step 3: Create a template for RTM (Add a column for each of the artifacts)
Step 4: Add Requirements from the Requirement Document (Add Requirements IDs and Description)
Step 5: Add Test Cases from Test Cases Document (Add Test cases corresponding to the Requirements mentioned in Step 4)
Step 6: Add Test Results and Issues, if any (Add the test results corresponding to test cases mentioned in Step 5 and any defects associated with them).
Step 7: Update the matrix – as and when required (Whenever there is a change in any of the artifacts, update the matrix so that it reflects the current health of the project)
Here is what a typical traceability matrix might look like in a simple Excel or Google Spreadsheet (although there would usually be more columns than this for various iterations over a project):
Excel can provide a quick, easy way to create an RTM matrix because of our familiarity with this tool, but the maintenance of RTM in Excel is a tedious job. It will take a significant amount of manual work to get the desired output. For example:
If the matrix doesn’t reflect accurate information, we won’t be able to use it to make data-driven decisions.
TestRail supports requirement traceability through comprehensive traceability/coverage reports available under the Reports tab.
These reports give an overview of our requirement coverage. We can also see all bug reports for our test cases at a glance and get a detailed matrix of the relationships between requirements, test cases, and bug reports.
Most teams prefer to manage their requirements/user stories/feature requests within their issue tracker such as Jira, wiki software, or dedicated requirement management tools.
When you start building your traceability report in TestRail, you should first link any of these requirements from your external tool in the References field available on TestRail Test Case and Test Result artifacts.
Important: the following reports require that you are using the References field to link to your requirement IDs. You can learn more about the References field here.
Now, let’s see how we can use reports like Coverage for References, Summary for References, and Comparison for References reports to show the coverage for requirements/user stories.
By default, the Coverage for References report will only include the issues that are already linked via the References field in TestRail.
To report on coverage across your entire list of requirements in any requirement management tool, export a list of the issue IDs from your project (in your requirement management tool) and upload them to a TestRail using the following steps.
Let’s say you are using Jira to track your user stories or requirements. Start by exporting a list of the requirements as issue from your Jira Project as a CSV.
Once exported, copy the list of issue IDs from the CSV export. Then open a new Coverage for References report, and under the Report Options > References tab, select the radio button “The following references only.”
Then paste the list of issue IDs (which are called “References” in TestRail) into the text area.
Then run the report! You will now see which Jira Issues have been associated with any test cases in TestRail.
If you believe you have already written the test cases you need to cover these requirements, but haven’t added the Reference IDs to the TestRail test cases themselves, you can do that now and rerun the report to check your updated coverage status. Otherwise, this report will help you identify which gaps you might currently have in your test plan and where you may need to write some additional test cases to achieve full coverage.
Coverage for References report will show the relationship between the references and the test cases. It will also show the test cases with or without references.
For example, see the screenshot below (References refer to the requirements we’ve linked to our TestRail test artifacts, and ID and Title refer to the test case ID and test case name respectively and blanks in the References column under IM-1 mean that IM-1 applies to all of those test cases.)
The Summary for References report will show a summary of defects you have found during testing and logged with your test results in TestRail. The report will show defect IDs you’ve entered during testing alongside your references and their test cases in a coverage matrix.
Note: The Summary for References Report will only show test cases and associated defects for test cases where you have specified a Reference as well.
If you entered test results with defect IDs for test cases and tests where you have not also specified a reference ID, that defect will not appear in this report.
Here, References mean the requirements, and ID and Title refer to the test case ID and test case name respectively. Also, the red color shows the failed status of the test case with defect ID next to it.
The Comparison for References report is perhaps the most important traceability report in TestRail and is the most similar to a traditional RTM.
In this report, you can compare the latest status of testing against your list of requirements to identify the overall progress of your testing, identify potential gaps in your testing underway, and triage any issues that have come up.
You will be able to see the latest status of tests run against various requirements from your project, as well as reveal if there are requirements that have yet to be tested. This way you can gain a more realistic sense of the coverage of your testing, evaluate the risks of deciding whether to keep testing or ship the product and reallocate resources as necessary to plug any coverage gaps.
You can have multiple runs/plans for the same suite/case repository active at the same time and this report will generate a matrix for the references/requirements/cases and test results. The Latest column shows the most recent test result (per test).
Here, References mean the requirements, and ID and Title refer to the test case ID and test case name respectively. Also, the test results and their corresponding color are shown with each test case and Reference ID.
Note: If multiple test cases are associated with the same requirement, the reference ID will not be copied for every row in this report to reduce clutter on screen. For example, in the above screenshot, test cases C3151, C3152, C3153, C3154, C3155, C3156, C3157, and C3158 are all associated with the same reference (issue): IM-1.
In the end, here are the top benefits of using TestRail Reports instead an RTM in Excel for reporting on traceability:
Help us improve this page!
What problem are you trying to solve?
We are thrilled to announce the launch of TestRail’s new brand. You have probably noticed a few new things already: updated colors, a refreshed logo, and our new “testrail.com” website. But our new brand goes beyond just colors and design....
Building QA into your SDLC is key to delivering quality. Here are the mistakes to avoid when building quality throughout the SDLC.
Organizations need conceptual and QA has crucial strengths to effect that change. Here are three winning action plans to change your QA culture and integrate it with the rest of your SDLC.