# Evaluating an AI pentest report: signs of real validation

Source: https://aipentestservices.com/guides/evaluating-an-ai-pentest-report/
Publisher: Excession Labs (aipentestservices.com)
Last updated: 2026-11-09

How to evaluate an AI pentest report: check for requests, read-back proof, controls, evidence labels and limits. Includes a scorecard with no numeric scores.

> **Short answer:** A good AI pentest report shows proof, not claims. Look for saved requests and responses with timestamps, read-back from a second account, controls that held, evidence labels such as Observed and Browser-observed, inferred paths marked apart, a limitations section, and reproduction steps. Scanner noise gives severity words and generic advice without evidence.

To evaluate an AI pentest report, check whether each finding comes with proof you can inspect. Real validation shows the request, the response, a read-back that confirms the effect, and controls that rule out false explanations. Scanner noise shows a severity word, a generic description and no evidence.

The scorecard below has no numeric scores, because a number hides the one missing item that matters.

## What does real validation look like?

Real validation proves an effect. A server can answer 200 and change nothing, and another can answer 403 and still change data. The report must show what happened.

Our own reports use these signals:

- Requests and responses are saved with timestamps.
- A write is proven by read-back: the tester changes a record, then reads it through another session.
- Controls run next to the exploit request: nonexistent identifiers, wrong record types, unsigned or wrongly signed cookies, another account's token, anonymous versus signed-in.
- Evidence has labels. "Observed" means reproduced, "Browser-observed" means shown in a real browser, and inferred paths are labeled apart.
- A "Limitations and open questions" section holds what was not proven.

The [example assessment](/example-assessment/) is a labeled sample run with a validated finding and a rejected false positive.

## Scorecard: what to look for in a report

> **Report evaluation scorecard:** Mark each row Present, Partly or Missing. Do not add numbers. Any Missing row in the first block needs an answer from the provider.
> 
> | Check | What good looks like | Red flag |
> |---|---|---|
> | **Proof** | | |
> | Requests and responses | Saved per finding, with timestamps | Screenshots only, or none |
> | Read-back | Change confirmed from a second session | "200 OK" as proof of a write |
> | Controls | Negative cases listed next to each exploit | Only the positive case |
> | Reproduction | Steps or read-only scripts that use test-owned IDs | "Try it yourself" with no steps |
> | **Honesty** | | |
> | Evidence labels | Observed, Browser-observed, inferred, kept apart | One confidence level for all |
> | Limitations | Named section: not tested, not proven | No such section |
> | Controls that held | Reported, so fixes keep them | Only failures listed |
> | **Usefulness** | | |
> | Severity scale | Written definition for each level | Severity words with no definition |
> | Fix guidance | Specific to your stack and the finding | Generic advice copied from a standard |
> | Scope and method | Assets, roles, dates and method stated | Scope missing or vague |
> | Evidence index | A list that maps findings to files | Evidence cannot be found |
> | Integrity | Key evidence with a SHA-256 manifest | Files can change unseen |
Ask about every Missing in the Proof block. A report that lacks read-back or controls has not shown that the issue is real.

## How do you tell scanner noise from a validated finding?

Scanner noise states a possible issue. A validated finding shows the issue working.

| Scanner noise | Validated finding |
|---|---|
| "Possible IDOR on /api/items/{id}" | Request as account A for account B's item, the response, and the read-back |
| "Missing security header" with no effect described | Header gap tied to a shown, exploitable path, or labeled as low-impact hardening |
| Same text for every customer | Text tied to your routes, roles and data |
| Severity from a plugin default | Severity from the written scale, with impact for your product |
| No negative tests | Controls show the check is not a false trigger |

[AI pentesting false positives](/guides/ai-pentesting-false-positives/) goes further, and [AI pentest agents vs DAST scanners](/guides/ai-pentest-agents-vs-dast-scanners/) compares scanners with agents.

## What should the report structure include?

Our reports have these sections:

1. Project details: scope, assets, severity scale.
2. Assessment method.
3. Safety constraints.
4. Evidence labels.
5. Executive summary with priority actions.
6. Findings at a glance.
7. Reproduction environment.
8. One section per finding: description, impact, verified results, reproduction, remediation.
9. Controls that held.
10. Remediation plan.
11. Limitations and open questions.
12. Evidence index.
13. Appendix of reproduction scripts.

We deliver HTML, PDF and Markdown, plus proof-of-concept scripts. [NIST SP 800-115](https://csrc.nist.gov/pubs/sp/800/115/final), published in 2008 and still listed as final, also asks for analysis and reporting that turns results into mitigation actions. Use the list to find gaps in any report.

## How do you read severity and impact?

Severity should follow a written scale, and impact should name your product. Our scale has four levels: Critical, High, Medium and Low. Each level has a written definition in the report.

Check three things for each rating:

1. The definition: Can you point to the sentence that puts this finding at High, not Medium?
2. The impact text: Does it say what an attacker gains in your app, such as reading another tenant's invoices? Or does it repeat a generic line about "data exposure"?
3. The conditions: Does the finding state what the attacker needs: an account, a role, a leaked token?

A rating with no definition is an opinion. Ask the provider for the rule behind it.

## What questions should you send back to the provider?

Send questions in writing and expect answers that point to evidence:

- Which finding has the weakest proof, and why is it in the report?
- Which checks did not run, and what stopped them?
- What did the controls show for the top three findings?
- Which paths are inferred, and what would confirm them?
- Which tests were bounded, and what did they cost?
- Who checked the evidence before the report was written?

Compare the answers with the report. If the report says one thing and the provider says another, trust the saved evidence.

## Is a report with many findings better?

No. Count does not measure quality. A long list of unproven alerts costs your team time. A short list of proven findings with controls gives you work you can start today.

Read the first three findings in full before you judge the rest. If they are well proven, the process behind the report is probably sound. If they are thin, expect the same across the report. Then check the section that lists controls that held. It shows the tester looked at working protections too, and it gives you a list to guard when you change the code.

## Who checked the AI's work?

Ask this of any AI-assisted provider. At Excession Labs, Incursion does the testing under tool rules. Excession Labs checks the evidence for each finding and writes the report. A task is done only when each finding links to its evidence.

Look for the same split in other reports: who ran the tests, who checked the proof, and who wrote the conclusions. If the report cannot answer, you cannot tell how much to trust it. [How agentic pentesting works](/guides/how-agentic-pentesting-works/) explains the pipeline.

## What to do after you read the report

1. Mark the scorecard. Send questions about Missing items.
2. Pick one High finding and replay its evidence, with authorization.
3. Check that the fix guidance fits your code.
4. Fix, then ask for a retest of each fixed finding.
5. Keep the controls-that-held list as a regression list for your own tests.

[How it works](/how-it-works/) covers a full engagement from scope to retest.

## Questions

### How long should a pentest report be?

Length is not a quality signal. A short report with proven findings beats a long one with unproven alerts. Check the ratio of evidence to claims, and check that the method and limits are written.

### Should a report use CVSS?

Not necessarily. Our reports use a four-level scale (Critical, High, Medium, Low) with a written definition for each level. CVSS vectors are not added by default. If your program needs CVSS, say so during scoping.

### What if a report has no findings?

A clean report is useful only if it shows what was tested and the controls that held. Check scope, method and limits. It shows that nothing was found in that scope, and it says nothing more about the system.

### Can I rerun the tests myself?

You can, with authorization and test-owned identifiers. Good reports include reproduction steps or scripts. Scripts should be read-only where possible. Do not run them against real users' data.

## Sources

- [NIST SP 800-115: Technical Guide to Information Security Testing and Assessment (2008, final)](https://csrc.nist.gov/pubs/sp/800/115/final)
- [OWASP Web Security Testing Guide](https://wstg.owasp.org/)