Why traditional QA metrics are misleading, what engineering leaders should measure instead, and why quality is ultimately about impact rather than activity.
The Problem With How We Measure Quality Assurance
Imagine two QA engineers working on the same product.
The first engineer writes 200 test cases, executes 500 tests, and reports 25 defects during a sprint.
The second engineer writes 30 test cases, identifies a critical architectural risk during a design discussion, challenges an ambiguous business requirement, and helps the development team prevent a potentially serious production incident.
Who performed better?
If your organization relies primarily on traditional QA productivity metrics, the first engineer probably looks more productive.
And that is precisely the problem.
For years, software organizations have measured QA performance through easily quantifiable activities: test cases created, tests executed, defects reported, and automation scripts developed.
These metrics are convenient. They fit nicely into dashboards, make sprint reporting straightforward, and give management something tangible to compare.
But convenience should never be confused with effectiveness.
We have become very good at measuring how busy QA engineers are, while often failing to measure how valuable their contributions actually are.
And in an engineering landscape increasingly influenced by automation and artificial intelligence, this distinction matters more than ever.
1. Test Cases Written Do Not Equal Quality Delivered
Let’s start with one of the most common QA productivity indicators: the number of test cases created.
Writing test cases is an important engineering activity. Good test design improves coverage, documents expected behavior, and supports repeatable validation.
But the number of test cases tells us very little about their actual effectiveness.
Consider two scenarios.
Scenario A: An engineer writes 150 test cases validating minor variations of the same functionality. Most cases cover straightforward positive flows, with limited attention to integration failures, security implications, or business-critical edge cases.
Scenario B: Another engineer writes 25 carefully designed tests covering critical business workflows, authorization boundaries, failure recovery, data integrity, and integration risks.
Which approach provides greater confidence?
The answer depends on the risks being addressed, not the number of test cases produced.
A well-designed test that exposes a serious financial calculation defect can be more valuable than hundreds of repetitive validations.
Even more importantly, sometimes the most valuable QA contribution doesn’t result in a test case at all.
It might be a question raised during refinement:
“What happens if the customer is charged, but the downstream payment service times out before confirming the transaction?”
That single question could trigger an architectural discussion about idempotency, reconciliation, and recovery mechanisms.
If the team addresses the risk before implementation, a potentially expensive production defect never exists.
No bug ticket. No failed test case. No impressive execution statistics.
Just effective quality engineering.
2. Test Cases Executed Do Not Equal Effective Testing
Another popular metric is test execution volume.
How many tests did the QA team execute this sprint?
How many passed?
How many failed?
These numbers are useful for understanding test progress and release readiness. But they become misleading when used as indicators of individual performance.
Executing 500 tests doesn’t automatically mean an engineer has contributed more than someone who executed 50.
A large test suite can provide a false sense of security if it repeatedly validates low-risk functionality while important failure scenarios remain unexplored.
And this becomes particularly relevant when we consider modern test automation.
If a task is repetitive, deterministic, and technically suitable for automation, why should we reward engineers for performing it manually at scale?
The purpose of automation is not simply to increase the number of tests executed.
It is to free engineering capacity for activities requiring human judgment.
Those activities include exploratory testing, investigating complex failures, evaluating architectural risks, understanding business workflows, and challenging assumptions.
Artificial intelligence takes this evolution even further.
AI-assisted engineering tools can already support test scenario generation, automation development, code analysis, test data preparation, and investigation of failures.
Tools such as OpenAI Codex can help engineers accelerate repetitive implementation work and spend more time on higher-value engineering decisions.
However, generating more tests faster does not automatically mean generating better tests.
AI can produce hundreds of test scenarios, but it cannot independently guarantee that those scenarios reflect the most important business risks or that their assertions are correct.
Human validation, technical understanding, and business context remain essential.
The objective should not be to maximize test execution. It should be to maximize the confidence gained from testing.
3. Bugs Reported Do Not Equal Engineering Impact
Of all traditional QA performance metrics, I consider defect count one of the most dangerous when used incorrectly.
Why?
Because people naturally adapt their behavior to what organizations reward.
If engineers are evaluated based on how many defects they report, they have an incentive to maximize the number of defects they discover and document.
But not all defects have equal business impact.
An engineer who reports 20 minor visual inconsistencies may appear more productive than someone who spends two days investigating a complex concurrency issue affecting financial transactions.
Yet the second engineer might have prevented significant financial losses, customer dissatisfaction, or regulatory exposure.
There is another, less obvious problem.
When we reward defect discovery without recognizing defect prevention, we risk encouraging a reactive quality culture.
Instead of collaborating with developers to prevent issues, QA engineers may become focused on discovering problems after implementation.
This reinforces the outdated perception that QA exists primarily to inspect completed software.
In reality, finding defects is only one part of quality engineering.
Preventing defects through better requirements, design discussions, technical reviews, and risk analysis can deliver considerably greater value.
This doesn’t mean defect counts are useless.
They can help identify unstable components, recurring failure patterns, or areas requiring additional attention.
But they should never become a competition between engineers.
The goal is not to discover more defects.
The goal is to deliver software with fewer defects that matter.
4. Goodhart’s Law and the Danger of Misleading Metrics
There is a well-known principle often summarized as:
“When a measure becomes a target, it ceases to be a good measure.”
This is commonly referred to as Goodhart’s Law.
It is particularly relevant to Quality Assurance.
When test case count becomes a performance target, engineers may produce more test cases than necessary.
When execution volume becomes a target, teams may prioritize easily executable tests.
When defect count becomes a target, engineers may focus on maximizing reported issues rather than reducing business risk.
When automation coverage becomes a target, organizations may accumulate large automated suites that are difficult to maintain and provide limited confidence.
In each situation, the metric begins influencing behavior in ways that can undermine its original purpose.
This doesn’t mean engineering organizations should abandon measurement.
Quite the opposite.
It means we need to become more deliberate about what we measure, why we measure it, and what behaviors those measurements encourage.
A useful metric should help teams make better decisions.
It should not simply make individual productivity easier to rank.
5. What Should We Measure Instead?
If traditional activity metrics are insufficient, what should engineering leaders focus on?
I believe a more mature quality measurement framework should consider five dimensions.
A. Production Stability and Customer Impact
Ultimately, software quality matters because customers depend on the product working correctly.
We should therefore pay attention to outcomes such as:
- Frequency and severity of production defects.
- Customer-impacting incidents after releases.
- Defect escape trends in business-critical functionality.
- Mean time to detect and recover from quality-related incidents.
- Recurrence of previously identified failure patterns.
These indicators help organizations understand whether their engineering practices are producing reliable software.
However, there is an important distinction.
Production quality is a shared engineering responsibility, not an individual QA KPI.
A production incident can result from architectural decisions, infrastructure limitations, incomplete requirements, development defects, deployment processes, or operational failures.
Holding an individual QA engineer responsible for every escaped defect would be both unfair and counterproductive.
Production metrics should primarily evaluate the effectiveness of the engineering system as a whole.
Individual contributions should be assessed through the actions engineers take to improve that system.
B. Risk Identification and Prevention
One of the most valuable capabilities of an experienced QA engineer is identifying problems before they become expensive.
This requires technical understanding, business knowledge, curiosity, and the confidence to challenge assumptions.
Rather than simply asking how many defects an engineer discovered, consider questions such as:
- Did they identify important risks during requirements refinement?
- Did they challenge assumptions that could lead to incorrect behavior?
- Did they identify missing negative scenarios or integration failures?
- Did their feedback influence a design or implementation decision?
- Did they help the team address a problem before it reached testing or production?
These contributions are not always easy to quantify.
And that is perfectly acceptable.
Not every valuable engineering contribution can or should be reduced to a numerical KPI.
A documented example of a prevented business-critical risk can be more meaningful than a dashboard showing hundreds of executed tests.
C. Involvement Across the Software Development Lifecycle
QA should not begin when development ends.
In mature engineering organizations, quality engineering starts much earlier.
QA engineers should participate in requirements discussions, architecture reviews, API contract design, acceptance criteria definition, risk assessment, and release planning.
Consider a QA engineer reviewing an API specification before implementation.
They identify inconsistent error handling, missing authorization checks, and unclear validation rules.
The development team addresses these issues before writing the code.
This is quality engineering at its best.
The engineer has reduced rework, improved the API contract, and potentially prevented multiple defects.
But traditional QA productivity dashboards may show no contribution whatsoever.
That is why we should evaluate not only testing outputs but also the engineer’s ability to influence quality throughout the SDLC.
Of course, attending more meetings is not the objective.
Meaningful involvement matters more than meeting participation.
The question is whether the engineer’s contribution improved decisions and reduced uncertainty.
D. Business Understanding and Product Ownership Mindset
Technical testing skills are essential, but they are not sufficient on their own.
Great QA engineers understand why a feature exists, who uses it, and what could go wrong from the customer’s perspective.
They don’t simply validate whether the implementation matches the acceptance criteria.
They also question whether those acceptance criteria adequately represent the business need.
For example, in digital banking, a transfer API might return a successful HTTP response while the underlying transaction remains in an inconsistent state.
A purely technical validation might consider the test passed.
An engineer with strong business understanding would investigate the transaction lifecycle, account balances, ledger entries, downstream processing, and customer-visible status.
The difference is significant.
One approach validates a response.
The other validates a business outcome.
This mindset becomes especially important in complex, regulated environments where technical correctness, financial integrity, security, and customer trust are deeply interconnected.
A QA engineer who understands the product can identify risks that would otherwise remain invisible.
E. Continuous Improvement and Engineering Enablement
Another important dimension is the engineer’s contribution to improving the effectiveness of the broader engineering organization.
Examples include:
- Developing reliable and maintainable automation frameworks.
- Reducing flaky tests and unnecessary maintenance.
- Improving CI/CD feedback loops.
- Introducing effective API testing strategies.
- Simplifying test data management.
- Improving observability and failure diagnostics.
- Mentoring engineers and sharing technical knowledge.
- Applying AI-assisted workflows responsibly to accelerate quality engineering.
These contributions can create value far beyond an individual sprint.
An engineer who improves an automation framework and saves the entire team several hours every week may deliver more long-term value than someone who simply adds another hundred automated tests.
This is the kind of engineering impact organizations should recognize.
6. A Practical Quality Engineering Scorecard
So how can we translate these principles into something engineering leaders can actually use?
I would recommend a balanced scorecard combining team-level outcomes with evidence of individual contributions.
|
Dimension |
What to evaluate |
|
Production Quality |
Critical defect escapes, incident severity, recurring failures |
|
Risk Prevention |
Significant risks identified early and mitigation influenced |
|
SDLC Contribution |
Quality improvements driven during refinement, design, and implementation |
|
Business Understanding |
Ability to validate business outcomes and challenge assumptions |
|
Engineering Improvement |
Automation reliability, faster feedback, reduced manual effort |
|
Collaboration |
Knowledge sharing, developer enablement, cross-functional problem-solving |
The important part is how these dimensions are used.
I would not recommend assigning arbitrary percentages to every category and calculating a final productivity score for each engineer.
That would simply replace one problematic measurement system with another.
Instead, engineering leaders should use these dimensions to structure performance discussions, identify growth opportunities, and recognize meaningful contributions.
Quantitative metrics should provide context.
Qualitative evidence should explain impact.
And the evaluation should account for the engineer’s role, seniority, scope of responsibility, and the complexity of the work.
A junior QA engineer and a principal quality engineer should not be expected to demonstrate impact in exactly the same way.
7. The Role of AI: More Output Is Not Necessarily More Value
Artificial intelligence introduces an interesting challenge to traditional QA measurement.
Imagine that an engineer uses an AI coding assistant to generate 100 automated test scenarios in an afternoon.
Another engineer spends the same afternoon reviewing a complex integration architecture and identifies a potential data consistency problem.
Under an output-based performance model, the first engineer might appear significantly more productive.
But what if 70 of those generated tests are redundant, 20 have weak assertions, and only 10 provide meaningful additional coverage?
And what if the architectural issue identified by the second engineer could have caused a major production incident?
AI makes it easier than ever to generate code, test cases, documentation, and other engineering artifacts.
That is a tremendous opportunity.
But it also makes activity-based metrics increasingly unreliable.
When the cost of producing engineering artifacts decreases, the value of judgment about which artifacts are worth producing increases.
This is why I believe the future of Quality Assurance will place even greater emphasis on critical thinking, technical depth, risk assessment, and business understanding.
AI should amplify the capabilities of good engineers, not become a mechanism for generating impressive but meaningless productivity statistics.
The question should not be how many tests an engineer generated using AI.
It should be whether those tests improved confidence, reduced risk, and contributed to a better product.
8. Quality Is a Team Outcome, but Individual Impact Still Matters
There is an important balance to maintain.
If we stop measuring individual activity, we should not automatically replace it with individual accountability for team-level outcomes.
That would create a different set of problems.
A QA engineer cannot single-handedly guarantee production stability.
They cannot compensate indefinitely for poor architecture, unrealistic delivery timelines, inadequate development practices, or organizational decisions that compromise quality.
Quality emerges from the combined effectiveness of product management, development, architecture, QA, DevOps, security, and leadership.
But this does not mean individual performance cannot be evaluated.
It means we should assess engineers based on their contribution to better engineering outcomes rather than treating them as isolated production units.
Did they identify important risks?
Did they help colleagues make better technical decisions?
Did they improve testing effectiveness?
Did they strengthen the team’s ability to deliver reliable software?
Did they demonstrate ownership and sound engineering judgment?
These questions provide a much more meaningful basis for performance evaluation than simply counting test cases or defects.
9. What QA Leaders Need to Change
Changing how we measure QA requires more than replacing a few dashboard indicators.
It requires a shift in leadership mindset.
As QA leaders, we need to stop treating test execution volume as a proxy for productivity.
We need to recognize engineers who prevent defects, not only those who discover them.
We need to create environments where QA engineers are encouraged to participate in technical discussions, challenge requirements, and influence architecture.
We need to invest in automation and AI not because they help us produce larger numbers, but because they allow engineers to focus on more meaningful work.
And perhaps most importantly, we need to stop confusing visibility with value.
Some of the most important contributions in software engineering are difficult to measure.
A critical risk identified during a conversation.
An architectural weakness challenged before implementation.
An ambiguous requirement clarified before development begins.
A production incident prevented by a well-timed question.
These contributions rarely generate impressive dashboards.
But they are exactly what mature quality engineering organizations should encourage.
Conclusion: Stop Rewarding QA Activity. Start Recognizing QA Impact.
Quality Assurance has evolved considerably over the years.
We have moved from predominantly manual testing toward automation, continuous testing, quality engineering, and increasingly AI-assisted development workflows.
Yet many organizations continue evaluating QA engineers using metrics inherited from an era when testing was treated primarily as a separate execution phase.
That needs to change.
Test cases written, tests executed, defects reported, and automation scripts developed can all provide useful operational information.
But they should not define the value of a QA engineer.
The future of quality engineering is not about producing more testing artifacts.
It is about combining technical expertise, engineering judgment, business understanding, and cross-functional collaboration to reduce risk and improve software outcomes.
Great QA engineers don’t simply find defects. They help engineering teams build systems in which fewer defects are introduced in the first place.
And sometimes, the most valuable QA engineer on your team is the one who prevented a critical defect that never appeared in your bug tracker.
That contribution deserves recognition.
Because ultimately, the real measure of Quality Assurance is not how much testing we perform, but how much confidence and value we create.