In today’s high-speed, on-demand world, customers expect seamless digital experiences. Websites and applications must be fast, responsive, and stable to meet these high-performance expectations and deliver a smooth user experience (UX).
Performance testing is essential to ensuring software performs reliably under different circumstances and conditions that mimic real-world usage. It evaluates how your website, app, or software handles various workloads, helping teams identify and resolve potential bottlenecks before they impact users.
There are several types of performance testing each designed to assess different aspects of system behavior. To better analyze and interpret test results, QA teams need to identify the most relevant performance metrics for each test type and set clear benchmarks for success.
In this article, we will focus on load and stress testing—two fundamental performance tests that provide critical insights into system stability and scalability. These tests help teams understand how their applications perform under expected and extreme conditions, ensuring a seamless user experience.
Load testing metrics
Load testing typically takes place before launching an application or website. It’s also important before releasing a new feature within an app or site.
To better explain how load testing works, let’s imagine that you want to test whether your website can handle a specific amount of users before a new product release or promotion. The last thing you want is for your website or app to crash during peak activity.
When analyzing the results from the load tests, some metrics can be useful for understanding performance. These include:
- Web server metrics: Evaluate how efficiently the web server processes user requests and manages traffic loads. These metrics depend on factors like server response times, request counts, and error rates.
- App server metrics: Assess an application server’s performance and efficiency by tracking CPU usage, memory utilization, and thread counts to identify potential processing bottlenecks.
- App health metrics: Measure key indicators such as application uptime, response times, and error rates to ensure stability under specific load conditions.
- Host health metrics: Monitor the overall performance of the system hosting the application, which includes physical or virtual servers. These metrics track CPU usage, memory consumption, disk input and output (I/O), and network bandwidth. Identifying resource constraints at the host level helps prevent performance bottlenecks that could degrade system stability.
- API metrics: Measure API throughput, latency, and error rates to ensure backend services perform reliably under varying traffic loads
Tracking these key metrics gives QA teams a clear picture of how a system is performing, making it easier to spot issues and fine-tune performance.
To keep things running smoothly, make load testing a regular part of your process—especially before major releases or updates. By integrating these tests into your CI/CD pipeline, you can catch potential problems early, long before they affect users.
Stress testing metrics
Stress testing evaluates how a system performs under extreme or unpredictable demand, pushing it beyond its normal operating capacity. This type of testing is crucial for identifying weak points that could cause failures and understanding how well the system can recover.
For example, major events like Black Friday or Cyber Monday can cause massive traffic surges, putting immense pressure on a website or application. Through stress testing, QA teams can pinpoint areas where the system struggles under heavy load and share these insights with development teams and stakeholders. With this information, developers can strengthen critical services to ensure they remain operational even under severe stress.
Key performance testing metrics for evaluating stress testing results include:
- Error rate: Measures how often errors occur during test execution and assesses the system’s ability to handle and recover from them. This includes issues like failed logins, notification errors, and system crashes.
- Endurance (soak) testing metrics: Track system performance over an extended period to detect memory leaks, resource depletion, or gradual performance degradation. This ensures the system remains stable and responsive under sustained load.
- Throughput: Determines how much data or how many requests a system can handle within a specific timeframe. It’s typically measured in transactions per second (TPS), requests per second (RPS), or data volume (e.g., MB/s or GB/s).
- Response time: Captures how long it takes for a system to process and respond to a request, measured in milliseconds (ms).
- Why response time matters: Fast response times are critical to user experience (UX). Measuring response time under different loads helps ensure the system remains performant and reliable, even during peak traffic.
Response time can be analyzed from multiple perspectives to better understand system performance:
- Average response time: The typical time it takes for the system to respond to requests, providing a general performance benchmark.
- Peak response time: The slowest response recorded under a specific test load, revealing worst-case scenarios during high traffic.
- Minimum response time: The fastest response recorded, showing the best possible performance under ideal conditions.
- Maximum response time: The longest response time observed across all tests, helping to identify potential bottlenecks and delays.
Spike testing
Spike testing evaluates how a system handles sudden or extreme increases in load or traffic. It helps determine whether the software can maintain stability during traffic surges and how quickly it recovers afterward.
Key metrics for evaluating spike testing results include:
- Response time under spike load: Measures how quickly the system responds when experiencing a sudden surge in traffic. A sharp increase in response time could indicate performance degradation under stress.
- Throughput during spike load: Tracks the number of requests the system can handle per second during the spike. This helps determine if the system can sustain high loads or if it becomes overwhelmed.
- Error rate under spike load: Monitors the percentage of failed requests during the spike. A high error rate suggests that the system struggles to handle abrupt increases in traffic.
- Recovery time under spike load: Measures how long it takes for the system to return to normal performance levels after the spike. Faster recovery times indicate better resilience and stability.
How to establish and calculate performance testing metrics
Understanding what you’re measuring, why it matters, and how to interpret the results is key to effective performance testing. Simply discovering that your application fails under a certain load isn’t enough—what matters is whether that failure threshold aligns with real-world conditions. If your system regularly encounters high traffic but isn’t designed to handle it, the test results lose value unless actionable steps are taken.
To make performance testing meaningful, set clear benchmarks that align with business goals and user expectations. Metrics should not only measure system stability but also provide insights that drive continuous improvements.
The best approach to defining performance metrics depends on your application’s specific needs. Start by identifying critical performance goals—for example, do you prioritize fast response times, high throughput, or system resilience? Then, establish fundamental metrics like response time, error rates, and throughput, which are essential for evaluating overall UX. Once the basics are covered, you can refine your analysis by incorporating more advanced metrics that provide deeper insights into system performance under different conditions.
Why percentiles matter in performance testing
When evaluating performance test results, averages alone don’t tell the full story. A system might have an acceptable average response time, but that doesn’t mean all users experience the same performance. Some users may face significant delays, especially under load spikes or stress conditions.
This is where percentiles come in. Instead of relying on averages, percentiles help gauge response time consistency by showing how different groups of users experience system performance.
How to use percentiles in performance testing: P90, P95, and P99
Percentiles are commonly used to assess real-world system behavior, especially when determining if a system can handle expected and unexpected load conditions. Here’s how they work:
- 90th percentile (P90): 90% of users experience response times at or below this value, while the slowest 10% wait longer. This helps identify general performance trends and whether most users get an acceptable experience.
- 95th percentile (P95): 95% of users have response times at or below this level, with only 5% experiencing slower responses. This is a strong indicator of overall user experience (UX) and a key metric for setting service-level objectives (SLOs).
- 99th percentile (P99): 99% of users receive responses within this threshold, but the slowest 1% face delays. This metric highlights outliers and helps identify potential bottlenecks or system limits that might impact high-value transactions or VIP users.
When to use percentiles in testing
Percentiles are particularly useful for:
✅ Identifying performance bottlenecks: High P99 values indicate severe slowdowns affecting a small percentage of users.
✅ Setting realistic benchmarks: Many performance teams use P95 or P99 instead of averages when defining acceptable response times in SLAs.
✅ Comparing different test scenarios: Tracking percentiles across different loads (e.g., normal vs. stress testing) helps understand system behavior under varying conditions.
By focusing on percentiles, QA teams can better predict real-world performance issues and ensure their system remains responsive for all users—not just the average case.
Metrics to use with caution
Some performance metrics may seem useful at first glance but can lead to misleading conclusions if not interpreted correctly. Instead of avoiding them altogether, it’s essential to understand their limitations and use them in the right context.
- Averages: While average response time is a common metric, it can be misleading. A small number of very slow or very fast requests can skew the average, making it seem like performance is stable when, in reality, some users experience significant delays. This is why percentiles (P90, P95, P99) are often better indicators of real-world performance.
- Standard deviation: This metric measures how much response times fluctuate within a test. While useful for assessing consistency, a high standard deviation means the average response time becomes less meaningful, as individual experiences may vary significantly.
- Metrics without context: CPU usage, memory consumption, and network bandwidth are important, but they don’t provide much insight on their own. For example, seeing high CPU usage isn’t necessarily a problem—unless it directly impacts response times or leads to degraded performance under load. Always tie these metrics to specific testing scenarios and real-world use cases.
Defining acceptance criteria in performance testing
Acceptance criteria set the minimum performance benchmarks a system must meet during testing. These criteria should be clear, measurable, and aligned with real-world expectations to ensure that software performs reliably under expected conditions.
When defining performance acceptance criteria, it’s important to determine what level of performance is acceptable for different scenarios. Percentiles play a key role here. For example:
- A P90 threshold may be acceptable for some applications—if 90% of users experience fast response times while only 10% see minor delays, this might be a reasonable tradeoff for a high-traffic e-commerce site.
- A P99 threshold is critical for mission-critical applications—if you’re processing financial transactions, running healthcare systems, or managing real-time communications, even a 1% failure rate could be unacceptable and require tighter performance guarantees.
To put this in perspective:
✅ If an e-commerce site has a checkout process that takes longer than 2 seconds for 10% of users (P90), that might be acceptable depending on user behavior.
❌ But if an event ticketing system experiences a 10% failure rate on QR code scans at the entrance (P90), that’s completely unacceptable, as it would lead to long lines and a bad user experience.
Setting the right thresholds
Acceptance criteria should always be based on business needs and user expectations. Consider these factors when defining them:
- What percentage of users should have a seamless experience? (e.g., P90 for general web traffic vs. P99 for financial transactions)
- What are the real-world consequences of slow responses or failures? (e.g., longer load times vs. outright transaction failures)
- How do performance tradeoffs impact usability? (e.g., slightly slower responses may be okay in some cases, but system crashes are never acceptable)
Performance testing metric examples
Understanding how to calculate key performance metrics is essential for analyzing test results effectively. By quantifying error rates, response times, and other key performance indicators (KPIs), QA teams can identify system weaknesses and track improvements over time.
Here are some fundamental performance metrics and how to calculate them:
1. Calculating Error Rate
The error rate measures the percentage of failed requests compared to the total number of requests handled by the system. This helps identify how often failures occur under different loads.
Formula:
\text{Error rate (%) } = \left( \frac{\text{Number of errors}}{\text{Total requests}} \right) \times 100
Example:
If a system processes 5,000 requests and 100 of them result in errors:
(100÷5,000)×100=2%(100 \div 5,000) \times 100 = 2\%(100÷5,000)×100=2%
A 2% error rate might be acceptable for some applications, but for critical systems, even a fraction of a percent could be problematic.
2. Calculating Average Response Time
Response time is a key metric for user experience, measuring how long it takes for the system to process a request.
Formula:
Average response time=Total response timesTotal number of requests\text{Average response time} = \frac{\text{Total response times}}{\text{Total number of requests}}Average response time=Total number of requestsTotal response times
Example:
If three requests have response times of 100ms, 120ms, and 150ms:
(100+120+150)÷3=123.33ms(100 + 120 + 150) \div 3 = 123.33\text{ms}(100+120+150)÷3=123.33ms
While the average response time is helpful, it’s important to also track variability (e.g., minimum, maximum, or percentiles) to understand performance fluctuations.
3. Calculating Throughput
Throughput measures how many requests or transactions the system can process within a specific time period, helping assess system capacity.
Formula:
Throughput=Total requests processedTest duration (in seconds)\text{Throughput} = \frac{\text{Total requests processed}}{\text{Test duration (in seconds)}}Throughput=Test duration (in seconds)Total requests processed
Example:
If a system processes 10,000 requests over a 5-minute test (300 seconds):
10,000÷300=33.33 requests per second10,000 \div 300 = 33.33 \text{ requests per second}10,000÷300=33.33 requests per second
This metric is useful for determining if the system can handle expected traffic volumes.
Performance testing best practices
To get the most value from performance testing and ensure accurate, actionable insights, it’s essential to follow key best practices throughout your testing process. Here’s a summary of the best practices we’ve discussed:
- Set clear performance goals and acceptance criteria: Define measurable benchmarks that align with your business objectives and user expectations.
- Use realistic test scenarios: Simulate real-world conditions by creating workloads that mirror typical user behavior and traffic patterns.
- Start early and integrate performance testing into the CI/CD pipeline: This helps identify potential issues early in the development cycle, reducing the risk of late-stage surprises.
- Focus on critical metrics first: Prioritize key metrics like response time, error rates, and throughput to evaluate core system performance before diving deeper into advanced metrics.
- Test under varied conditions: Include different scenarios, such as peak loads and stress tests, to understand how your system behaves under various demands.
- Monitor system resources: Keep an eye on CPU usage, memory consumption, network bandwidth, and other resources to detect potential bottlenecks.
- Leverage trends and patterns: Track performance results over time to identify trends and recurring issues that may impact user experience (UX).
- Continuously refine your testing process: Use the insights gained from testing to optimize your performance strategy and address emerging challenges.
- Use reliable performance testing tools: Proven software, like TestRail, can streamline test case management, track results, and help QA teams stay organized and efficient.
Simplify performance testing with TestRail
Managing performance testing effectively requires the right tools to track results, analyze trends, and optimize test coverage. TestRail provides a centralized test management solution that helps QA teams streamline their performance testing efforts and track critical performance metrics.
Image: Manage all of your manual, exploratory, and automated tests in one place to gain full visibility into your testing.
With TestRail, teams can:
- Log and track performance metrics such as response times, error rates, and throughput within structured test cases.
- Integrate with automation frameworks like Selenium, Cypress, and JUnit to capture and analyze test execution data in real time.
- Automate performance test result collection using TestRail’s API and command-line integration (TRCLI).
- Generate detailed reports that consolidate key performance data, making it easier to track trends and identify bottlenecks.
- Link performance issues to defects and requirements, ensuring that slowdowns or failures are documented and resolved.
By centralizing performance test data and enabling real-time insights, TestRail helps QA teams evaluate performance trends, detect anomalies, and optimize system behavior before release.
Want to simplify performance testing and metric tracking? Try TestRail free for 30 days and see how it can improve your test management workflow.
