This is guest post by Ostap Elyashevskyy
If you are building your QA process from scratch or improving your existing QA process, choosing the right test management tool is crucial to help you find bugs, manage test cases, and ultimately, streamline the software development process.
In this post, I will share the experience that our team at ELEKS had searching for a test management solution that would work for our team, some alternatives we evaluated along the way, and why we ultimately chose TestRail.
Some of the most common challenges you may experience at the start of a software development project include:
It is easy for testing work to be invisible. I’ve seen so many occasions where testers do all the work, but the results stay locked up on their minds or on their machines, not helping the project at all. Testing is critical in any software project, but only if the results are communicated effectively and quickly.
QA is not only about running tests. It’s just as important to generate visibility for your team via good reporting and dashboards. Ensuring visibility means understanding the answer to questions like, “Who is testing?”, “What functionality is being tested?”, “What is the scope of testing,” and “When will we finish all testing activities?”.
Oftentimes, when teams do not have the proper test management tools in place, they experience poor usability throughout their projects. Many projects track their test cases and checklists in spreadsheets. While spreadsheets are an option for smaller projects, they are unsustainable for larger projects with multiple engineers. With spreadsheets, updating and managing test data oftentimes becomes a time-consuming and error-prone nightmare.
If you choose a test management tool that does not align with your organization and its practices, you are at a high risk of wasting time and money. When a test management tool is not used to its full capabilities, it is most likely the wrong tool for the job. The wrong tool can wind up simply as a means of test case storage or another wasted tool in your company’s collection of unused software.
Teams often encounter two core problems with their test cases: they don’t have enough contextual information, or they are written in a way that makes them hard to reuse.
For example, test cases exist but there is no information about priorities, versions, configuration, references to requirements, or other important information. This makes it difficult to prioritize the right tests, test accurately, and report on the results of testing in a way that’s meaningful to the rest of the team.
Check whether your test management tool integrates with your existing tool stack. In particular, can it integrate with your bug or issue-tracking software, such as Jira? The advantage of such integration is that users can easily link bugs to test case runs and benefit from advanced traceability. Lacking the proper integrations can prevent test cases from being executed in the scope of a feature.
The goal of traceability is to help you plan and manage testing activities (including defect management) to minimize risk better. A lack of traceability can create problems throughout the software development lifecycle (SDLC). For example, lack of traceability can make it difficult to know the impact of changes, resulting in inconsistencies that can derail your testing progress, waste time, and deplete resources.
Having an established test management tool does not mean your problems with traceability, visibility, and documentation are automatically solved. I experienced this firsthand when using spreadsheets as a test management tool and TestLink as our test management tool before switching to TestRail.
Let me elaborate — although TestLink is a free open-source test management tool, it does not offer feature advantages over other test management systems. TestLink has poor usability, and the Jira integration is one-directional. TestLink’s Jira integration only allows you to:
Other limitations you may experience with TestLink include:
If you want to work with the most user-friendly and powerful tool that fits the needs of your team, I recommend you to install trial versions of the tools you are scoping.
Personally, I tried both Zephyr and TestRail in depth and chose TestRail as our test management tool. For better visualization of the pros and cons, I created a table that summarizes my experiences and will hopefully help you to choose the most optimal tool for your team.
Zephyr pros:
Zephyr cons:
Ultimately, because of poor usability, lack of visual dashboards, and a pricing model that did not fit our needs, Zephyr was not a good fit for my team.
We chose TestRail as our test management solution because of the following benefits:
TestRail integrates with Jira in many different ways. You can easily push bug reports to Jira, look up issue details, jump to Jira’s New Issue form and more. Moreover, teams can extend TestRail’s defect plugin for Jira, enabling them to adapt the integration to their Jira customizations if needed.
The Jira integration for TestRail provides better visibility and traceability than other test management tools.
It also allows you to easily answer questions like:
TestRail supports two-way integration with Jira so you can:
Here are the two steps to configure the Jira test management integration for TestRail:
TestRail integrates with Jira Cloud, Server, and Data Center. Here is TestRail’s Jira integration documentation with setup instructions for each platform:
Let’s check the main features which can help you to improve visibility:
In the picture below, test cases are associated with the Jira subtask, which was created to track test design activities (in my case).
If you open the “QA – Test Design” subtask with id: DESK-10 (in my case) in Jira and go to the “TestRail: Cases” section, you can see the associated test cases with it:
If you create a subtask or another issue type in Jira for testing purposes to run some tests (Ex: “QA – testing feature”), you can create a test run in TestRail and associate it with a Jira issue (subtask).
Now, if you open this subtask in Jira, the result of the test run will appear inside the Jira issue, so you don’t need to open TestRail for it (but you can open test results in TestRail as well):
At ELEKS we use Python API and our test automation engineer, Maksym, implemented a wrapper over TestRail API which adds extra functionality. So, if you are using a Python client – you can find original Python API bindings or Maksym’s API bindings here.
There are many other TestRail features that we found useful for test management.
Shared steps allow you to reuse common steps across test cases and improve the maintainability of test cases (not to mention saves you time!).
Reusability is achieved by “importing” shared steps. Be sure to carefully consider your naming convention for shared steps because you are likely to have hundreds of shared steps. Using a pattern for naming shared steps (for example, <functionality or page> – <step description>) will help you to navigate through steps and import them easily.
If any changes are needed, you can update it in a single place and other test cases will reflect the changes.
TestRail’s dashboards and charts represent the current status of testing. You can share them at sync meetings or send them regularly to give stakeholders a quick overview of testing progress.
The level of detail in the reports can also be configured in the “Reports” section.
Reports are comprehensive and can be generated according to 15 predefined templates sorted by categories: “Cases,” “Defects,” “Results,” “Summary,” and “Users.” You can use templates from the “Summary” section to generate a universal report that can be helpful to everyone from QA engineers to project managers and product owners.
If you are a QA lead – this feature allows you to distribute testing across the team and track the progress.
You can customize your test case, result status fields, priorities, templates, and other fields. In our case, we added additional result statuses below to provide more information about the test run:
When time is limited, it can be cumbersome to describe all steps in a test case. To save time, you can use a simplified template for test cases or even an exploratory session with a description of the functionality you want to verify.
TestRail’s two-way Jira integration provides you with the ability to see the execution status from TestRail and Jira and allows you to preview a Jira issue when referenced with TestRail result or test case.
TestRail’s usability and intuitive user interface allow for quick onboarding and the flexible licensing model allows you to pay as you go. This approach worked well for us as we started with 2 users and then scaled up to over 20 active users in the system, including developers who use TestRail daily as their source of truth for test cases, data, results, and overall progress.
TestRail allowed us to streamline our test processes, leverage integrations with Jira, and improve visibility and usability. Ultimately, by using TestRail as our dedicated test management solution, our QA team was able to gain visibility into testing, scale QA more efficiently, and accomplish more testing.
This post covers how to show test coverage, manage requirements traceability, and create a traceability report in Jira.
Learn about the pros/cons of using Jira as a test management solution, alternatives, & how teams that use Jira manage their testing.
a list of the top 5 QA metrics that provide valuable metric data that can improve the software application’s quality continuously.