Every QA team reaches a point where the test suite has outgrown the time available to run it. Regression suites grow sprint by sprint while release windows don’t. And with AI-assisted development accelerating how much code ships, the number of test cases teams are expected to cover keeps climbing.
The response, for most teams, is a manual prioritization call before every run: which tests matter most right now? Who has context on the recent changes? What failed last time? It works, but it’s slow, inconsistent, and it relies on whoever happens to be in the room.
TestRail 10.5 introduces AI Test Prioritization—a smart execution feature that answers that question automatically, using the execution data already sitting in your TestRail instance.
How It Works
AI Test Prioritization scores every test in a run from Very High to Low priority using a combination of machine learning and semantic AI. It draws on up to 60 days of your execution history, analyzing failure rates, defect frequency, and flaky test patterns to surface the tests most likely to matter.
Select a run, choose your look-back window (7, 30, or 60 days), and the AI returns a prioritized list with an explanation for every ranking. Tests that failed repeatedly sit at the top. Tests tied to multiple defects rank higher. Consistently passing, stable tests move down. What was previously buried in months of execution logs becomes immediately actionable.
You can prioritize between 10 and 250 tests in a single run, and priority rankings are available via API for CI/CD integration, so the same intelligence that helps your team triage manually can also drive automated pipeline decisions.
When Historical Data Isn’t Enough
Machine learning scoring works well when you have history to draw on. But not every test run does.
Testing a new feature from scratch? Working on a product with a freshly created test suite? No historical failure patterns exist yet, so a purely data-driven model would have nothing to work with.
This is where semantic reasoning takes over. Rather than falling back on a random or arbitrary order, AI Test Prioritization uses the contextual information already inside your test cases—like feature descriptions, requirement context, linked Jira tickets, impacted components, and test intent—to infer which tests are likely to matter most.
Even without a single previous execution, the AI can reason about risk based on what the tests are actually covering, ensuring that new features aren’t a blind spot.
Smart Context for the Current Release
Historical data tells you what failed before. It doesn’t always tell you what matters right now.
That’s where label boosters and natural language instructions come in. Before prioritizing a run, you can tell the AI exactly what the current release needs. Want to weight security-labeled tests more heavily before a compliance audit? Need payment flows ranked above account management for this particular sprint? Just say so in plain English. The AI combines that instruction with the historical scoring to produce an execution order that reflects both your data and your current priorities.
Teams can configure custom boosters using any labels already applied in their TestRail projects, with no additional setup required.
Transparent Reasoning, Full Manual Control
Every priority decision comes with an explanation. You can see exactly why a test ranked Very High, whether it’s a high historical failure rate, multiple associated defects, flaky behavior, or semantic context, and override any ranking manually if you disagree with the AI’s assessment.
This matters because trust in AI tooling builds incrementally. Showing the work rather than just producing a result means QA leads can sanity-check the output, catch edge cases where historical data doesn’t tell the full story, and make informed decisions about when to follow the ranking and when to adjust it.
Where This Fits in the TestRail AI Story
TestRail 10.5 completes a testing lifecycle that TestRail AI has been building toward since 9.5.
- AI Test Case Generation (9.5) helped teams create test cases faster from requirements.
- AI Test Script Generation (10.2) turned those cases into automation scaffolding.
- The AI Evaluation Template (10.3) gave teams a structured way to test AI features themselves.
- And now AI Test Prioritization (10.5) answers the question that comes up every single run: what do we actually execute first?
The result is AI that covers the full workflow: generate, automate, evaluate, prioritize. Each feature addresses a different point in the testing process, and each one uses the data and structure already inside TestRail rather than requiring teams to adopt new tools or change how they work.
Get Started with TestRail 10.5
AI Test Prioritization is rolling out now, and will be available for all TestRail Cloud customers by early July. It uses the same AI credit system introduced with AI Test Script Generation in TestRail 10.2.
To get started:
- Read the TestRail 10.5 Release Notes
- Register for the Launch Webinar on July 7 at 11 AM EDT / 5 PM CEST
- Take the free TestRail Academy Course to master the new feature and earn a LinkedIn-ready certification
