There are some approaches and practices that will help your team get the most value from TestRail. In a recent webinar, TestRail product manager Simon Knight shared his “pro tips” to help test managers/test leads optimize testing workflows for any development approach: waterfall, agile/scrum, or DevOps/continuous.
The key points from the webinar are summarized in this article. Use the links below to jump to the topic that most interests you. You can also watch the full webinar on-demand.
Test management is “the set of testing activities necessary to support the successful release of a product, including test planning and design, test execution, test function administration (such as test maintenance), and reporting. TestRail is designed to support all of these activities.
As a test case management system, TestRail does not directly support requirements for the purposes of linking them to your test cases. But, it does integrate with a significant number of requirements management tools, such as Jira, Assembla, or GitHub (yes, even if you’re using Word or Excel to manage requirements). Use the TestRail test cases “references” field as a direct link to your requirement. TestRail test runs can also have references. TestRail has dedicated plug-ins for Jira and Assembla for extended requirements support.
TestRail offers several templates for setting up your test cases. You can also set up your own custom templates if none of the default templates meet your needs.
If you’re following a sequential, waterfall-style development approach, then you might want to use the Test Case (Steps) template shown above to set up your test cases. The Test Case (Steps) template offers flexible test script definition, and allows you to decompose scripts into steps and expected results. Each step gets its own status at runtime. In addition, this template is ideal for business or user acceptance testing (BAT/UAT).
If you’re using more of a Scrum or agile approach, then the Exploratory Session template may be a better fit. This template offers a loose framework for test definition and aligns with various exploratory approaches. The approach that you want to follow with exploratory testing is to identify a set of exploratory testing coverage, execute that initial set of session-based tests, and then iterate as you learn more about the product. For additional information, there are several resources you can refer to, such as Exploratory Software Testing by James A. Whittaker, Session-Based Test Management by James & Jon Bach, and Thread-Based Test Management by James Lyndsay.
For a DevOps/Continuous approach, the preferred test case template is Test Case (Text). That’s because your release artifact is generally considerably smaller than in the other approaches, perhaps consisting of only a single pull request. The Test Case (Text) template works well as a placeholder for automated test activities. You can use the TestRail API to pull/push information from and to TestRail.
The general principles of managing test cases in TestRail are as follows:
Once you have your test cases created, you’re ready to start creating and executing test runs. The specifics vary somewhat based on your development methodology.
If your team is following a sequential/waterfall approach to development, you have several options to organize your test runs (using the Test Plans & Milestones capabilities of TestRail):
For an agile/Scrum approach, consider organizing your test runs as follows:
For a DevOps/Continuous approach, use the TestRail API to create runs and report test automation results. You only need two endpoints: the add_run endpoint, passing in the IDs of the test cases that you want to execute. Use the add_results_for_cases endpoint to add the results for the cases in your test run. You can also use those endpoints to pass in results for milestones, and add references to your requirements in those runs.
When setting up your automated testing test runs, it’s helpful to do the following:
You can choose to get your automated results in real-time (by refreshing the page) or use a bulk API endpoint to get all of your test results in a single pass, which is an easier and more performant way to use the TestRail API for your automation.
From a functional perspective, test plans and milestones to organize your testing efforts are entirely optional. You can use TestRail with only test cases and test runs. But organizing your runs into plans and milestones has benefits for your status and reporting efforts:
TestRail does the report configuration for you, by automatically populating the summary reports based on the TestRail view that you used to navigate to that report.
To this point, the webinar has addressed how you can think about test management in terms of workflows depending on the methodology that you’re using: waterfall, agile/Scrum, and continuous. While there are similarities, the feedback loop: you’ve got requirements that link to test cases, the test cases are linked in runs organized as needed for your specific context; and then grouped under a release version or milestone. Generally, less organization in continuous due to the shorter feedback loops.
In summary, here’s the TestRail approach to test management:
If you’re mainly following a sequential/waterfall approach, here are some test management tips:
If you’re mainly following an agile/Scrum approach, consider doing the following:
Finally, for a DevOps/Continuous approach:
If you haven’t yet tried TestRail for test case management, start your free trial today!
We are thrilled to announce the launch of TestRail’s new brand. You have probably noticed a few new things already: updated colors, a refreshed logo, and our new “testrail.com” website. But our new brand goes beyond just colors and design....
By integrating Global App Testing with TestRail, you can launch unscripted and scripted tests and receive results in as little as 15 minutes.
Building QA into your SDLC is key to delivering quality. Here are the mistakes to avoid when building quality throughout the SDLC.