More Findings Don't Mean Better Penetration Testing

A penetration testing report with 200 findings might look impressive. But what if some findings cannot be reproduced, several describe the same underlying weakness and the most serious exposure is buried halfway down the list?

The number of findings makes a poor measure of security. What matters is whether the testing reveals exploitable weaknesses, demonstrates their potential impact and gives the organisation enough evidence to decide what to fix first.

As AI becomes part of security testing workflows, this distinction deserves more attention. Automation can help teams examine larger sets of assets and accelerate parts of an assessment. Yet faster testing only creates value when the results are reliable.

A fresh reminder from AI security testing

On 20 October 2026, the OWASP GenAI Security Project published version 1.0 of its GenAI Red Teaming Manual. Its eight-phase methodology covers planning, reconnaissance, attack surface mapping, exploitation, persistence, impact assessment, reporting and remediation. The manual emphasises reproducible findings and measurable outcomes when evaluating generative AI systems. OWASP's announcement offers a timely reference for organisations building more disciplined testing processes.

The manual focuses on generative AI systems, including applications and agents. It isn't a universal methodology for every infrastructure penetration test. Its emphasis on evidence and repeatable results, however, reflects a principle that applies across security assessments: a claim needs to stand up to verification.

NIST's Technical Guide to Information Security Testing and Assessment also covers the planning and execution of technical tests, analysis of findings and development of mitigation strategies. It discusses the benefits and limitations of different testing techniques, a useful reminder that every method has boundaries.

What a useful AI-assisted test should deliver

A credible assessment should give security teams more than a list of possible weaknesses. Each material finding should have enough supporting evidence for a technical team to investigate and act.

Look for four qualities in the output:

• Reproducibility: Can an authorised tester repeat the steps and observe the same result?

• Demonstrated impact: Is there evidence of what an attacker could actually access, change or disrupt?

• Clear prioritisation: Does the severity reflect business exposure and realistic exploitability rather than a generic score alone?

• Remediation and retesting: Does the report explain how to address the issue and verify that the fix works?

These checks matter because automated testing tools vary in what they detect and how reliably they detect it. NIST's 2023 SATE VI Report: Bug Injection and Collection examined static-analysis tools and found that detection performance varied across test cases, bug classes and levels of complexity. The study concerned code analysis rather than penetration testing, but it illustrates why organisations should evaluate tools against their own requirements instead of assuming that automation performs equally well in every situation.

Where AI-assisted penetration testing fits

AI-assisted penetration testing can help expand assessment coverage and reduce manual effort in parts of the testing process. Its value depends on how well the workflow defines scope, gathers evidence, validates findings and communicates risk.

Organisations should establish authorised testing boundaries before a run begins. They should also decide what constitutes a confirmed finding, what evidence must be retained and when a human tester needs to review a potentially disruptive action. These controls help keep faster testing accountable and its results useful.

For security leaders, the question should be less about how many findings a platform produces and more about what the team can confidently do with them. A shorter report with verified attack paths and clear remediation steps may be far more useful than a lengthy report full of untested possibilities.

Bronyx AI, ITSEC's AI-assisted autonomous penetration testing platform, is relevant to organisations exploring how AI can support their security testing workflows. The practical starting point is to assess how a testing approach fits your environment, authorisation requirements and evidence standards, then evaluate the quality of its results.

Explore Bronyx AI at bronyx.ai.

References

1. OWASP GenAI Security Project, GenAI Red Teaming Manual Version 1.0, 20 October 2026. Read the manual

2. National Institute of Standards and Technology (NIST), Technical Guide to Information Security Testing and Assessment, 30 September 2008. Read the guide

3. National Institute of Standards and Technology (NIST), SATE VI Report: Bug Injection and Collection, 14 June 2023. Read the report