Using TestRail for Your Regression Testing

By default TestRail comes with a set of fields that are useful for structuring, organizing, and compartmentalizing test cases. But you can customize the fields to record any additional information that your team needs.

This is a guest posting by Nishi Grover Garg.

Agile teams constantly face an overload of regression testing. As sprints progress and new features get built into the software, agile testers are faced with the challenge of testing all these new features while ensuring nothing from the previous sprints breaks as a result of the code changes and defect fixes. This piling regression can easily become a nightmare if not planned for properly in the beginning.

Planning for Regression

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

Every team will have a different approach to regression. The intensity of regression tests needed will be decided based on factors like the complexity of the system, the number of interfaces with other systems, the size of the change being made, and the safety-critical nature of the software. The agile team’s regression strategy must be decided at the onset of the project, and effort and time allocation must be done in the release test plan. Further, every sprint test plan must have an allowance for a regression effort in addition to the sprint-level testing to be performed.

Regression strategies vary from exhaustive to selective, based on the nature of the project and the context of the current release cycle.

Teams may decide on a “retest all” strategy, where you rerun all test cases for the software or feature at hand as a part of regression.

Test selection is where you select some of the test cases for regression from the entire pool of test cases based on the functionality covered or modified. This decision may be made by the test manager or the testers, depending on their knowledge and experience.

Prioritized regression is where you select the test cases for regression based on their priority in the context of the cycle. For example, for a major release cycle, you may decide to rerun all high- and medium-priority test cases for the entire application. On the other hand, for a hotfix or a patch release, you may decide to run only high-priority test cases for the modified feature during regression.

Deciding the regression strategy for your agile team is the most important step in the initial phases of the project.

Performing Regression Using TestRail

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

TestRail is a test case management tool. It makes regression testing easy by providing you with a repository for all of your tests, ways to track changes to those tests, results associated with the tests, and ways in which to plan, sequence, and execute all of your tests during the various phases of your approach, including automation integration if this is a part of your strategy.

By default TestRail comes with a set of fields that are useful for structuring, organizing, and compartmentalizing test cases. But you can customize the fields to record any additional information that your team needs.

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

Test case history is available to see how many times tests have been executed, what the results were each time, and any additional information, like attachments or defects, found.

All recent test results for a test case also can be found at a glance in the test results tab.

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

When starting test runs for a test suite, you can decide to include all or selected tests for that cycle. So whenever you need to test a new version of your software, you can simply start a new test run against your existing test cases and include a number of tests from that test suite based on your regression strategy.

Selection of Regression Test Cases

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

TestRail allows testers to manually select test cases or entire test sections based on the changed areas or functionalities that will get included in the test run.

It also allows testers to select the test cases for a test run based on multiple filters, like priority of test case, type, title, or area. For example, if you do not have enough time for revisiting all test cases of a particular area, you can select test cases of the highest priority only. Or let’s say you would like to run only the test cases previously marked as regression tests — you can select the filter for Type as “regression.” And a combination of both of these filters would select all regression tests with a high priority for this test run.

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

This allows testers to reuse the test cases already present for their regression cycles. There is no need for creating a separate regression test suite or duplicating any tests for these types of runs.

Performing Regression on Different Platforms

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

Configurations and operating systems can be added in TestRail for repeating the test runs on different platforms. You can even start multiple test runs for different platforms, including operating systems or web browsers you need to test against.

TestRail automatically replicates the test run for the selected number of configurations and provides a comparison of run results on different platforms once executed.

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

Monitoring and Reporting your Regression Test Cycle

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

TestRail comes with built-in reporting options, and many of the commonly needed metrics for agile teams are available within the interface. It also integrates easily with defect tracking tools like Jira, Redmine, and Bugzilla, so any defects found during testing can be specified with the failed test case itself.

All test changes, results, comments, and attached files are displayed in a single place, maintaining the history of the test case.

Monitoring the progress of your test run cycle is made easy by the visual representation on the test run page, which shows the number and percentage of tests executed, along with pending tests, with their results status. Testers can easily mark the results of the tests executed directly from this page, too.

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

A test manager can create print and PDF reports for the test cycle in progress in one click, saving time and effort for the collection and collation of the information in order to share it with stakeholders. They can also export the report as various spreadsheet file types to further analyze the data.

TestRail also has a forecasting feature that uses your historical time data and entered estimates to predict how much time your test will take to execute and how much of the remaining effort is needed for your test run. This is represented in the familiar burn-down chart format on the progress page, along with a projected completion date.

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

Extensive reporting options are also available, along with ready templates to generate summary reports, comparison reports for multiple test runs, coverage reports and defect charts.

Regression Testing, Agile Teams, Planning for Regression, Performing Regression Using TestRail, Performing Regression, Regression Test, Regression Strategies. TestRail.

With all these features, TestRail can ensure smoother planning and execution of your regression testing, with complete transparency of progress and test results of each test case. Tools like TestRail can take the overhead of management away while keeping your team completely focused on their quality goals.

To learn more about TestRail, you can watch this video series to get a better feel for TestRail’s capabilities.

Nishi is a consulting Testing and Agile trainer with hands-on experience in all stages of software testing life cycle since 2008. She works with Agile Testing Alliance(ATA) to conduct various courses, training and organizing testing community events and meetups and also has been a speaker at numerous testing events and conferences. Check out her blog where she writes about the latest topics in Agile and Testing domains.

In This Article:

Try a 30-day trial of TestRail today!

Share this article

Other Blogs

Accessibility Testing in Action: Tools, Techniques, and Success
Software Quality, Agile, Automation, Continuous Delivery

Accessibility Testing in Action: Tools, Techniques, and Success

In today’s digital world, accessibility is essential—not just a nice-to-have. December 3rd, the International Day of Persons with Disabilities, reminds us how crucial it is to create inclusive digital spaces. For over 1 billion people living with disabilities,...
User Acceptance Testing (UAT): Checklist, Types and Examples
Agile, Continuous Delivery, Software Quality

User Acceptance Testing (UAT): Checklist, Types and Examples

User Acceptance Testing (UAT) allows your target audience to validate that your product functions as expected before its release. It ensures that you correctly interpret the requirements, and implement them in alignment with what users want and expect. What is...
Complete Guide to Non-Functional Testing: 53 Types, Examples & Applications
Software Quality, Performance, Security

Complete Guide to Non-Functional Testing: 51 Types, Examples & Applications

Non-functional testing assesses critical aspects of a software application such as usability, performance, reliability, and security. Unlike functional testing, which validates that the software functions in alignment with functional requirements, non-function...