# AI pentesting false positives: why agents over-claim and how to validate

Source: https://aipentestservices.com/guides/ai-pentesting-false-positives/
Publisher: Excession Labs (aipentestservices.com)
Last updated: 2026-11-09

Why AI pentest agents report bugs that are not real, the evidence standard that stops it, and a validation checklist you can apply to any AI-generated finding.

> **Short answer:** AI pentest false positives are findings that an agent reports as real but that have no real effect. They arise because a language model reads a response and judges it, and a success message looks like proof. The fix is an evidence standard: prove the effect by reading state back, run controls beside each exploit request, and have a human check every finding.

An AI pentest false positive is a finding that an agent reports as real but that has no real effect on the target. A bare success response is not proof. A server can answer `200 OK` and change nothing, or return `{"ok": true}` while it ignores the request. If a report treats that answer as evidence, the finding may be false.

## Why do AI pentest agents over-claim?

Agents over-claim because a language model judges text, and a response that looks like success reads as success. Four causes recur.

1. Success-shaped responses. Many APIs return `200` or a friendly message for a request they quietly ignore. The model reads the message and concludes that the action worked.
2. Echoed input. The server reflects the attacker's value back. The agent sees its own payload in the response and calls it injection or stored data.
3. Public data. The agent fetches another user's record and calls it an authorization failure. The record is public by design.
4. Pressure to finish. An agent that is asked for findings tends to produce findings. A task with no evidence rule rewards a confident claim.

Each cause comes from a design that lets the model decide alone that a test passed.

## What evidence standard stops false positives?

A finding needs evidence of a real effect. Our rule is that a write is proven by reading the changed record back through another session. A "success" response alone does not count. In Incursion, a task is done only when each finding links to its evidence.

[OWASP API1:2023](https://owasp.org/API-Security/editions/2023/en/0xa1-broken-object-level-authorization/) describes attackers manipulating "the ID of an object that is sent within the request". Proof is a response or stored state that shows an object the caller should not reach.

| Claim type | Weak evidence (do not accept) | Strong evidence |
|---|---|---|
| Read another user's data | `200` status | Response body holds a record that belongs to the other account, and a control shows the owner can see it and a third party cannot |
| Change another user's data | `200` or `{"ok":true}` | Owner session reads the record back and shows the new value |
| Delete a record | `204` status | Owner session requests the record and gets `404` |
| Injection | Payload echoed in the response | Payload runs in a real browser, or changes server behavior in a way a control does not |
| Missing authentication | One request without a cookie works | Same request with and without a valid cookie differ in a way that shows the protection is absent |

## What are controls, and why run them beside the exploit?

A control is a second request that should fail or behave differently, sent next to the exploit request. It shows that the test can tell a real effect from noise. Without controls, a positive result might come from a server that accepts everything.

The controls we run beside exploit requests are:

- nonexistent identifiers;
- wrong record types;
- unsigned or wrongly signed cookies;
- another account's token;
- anonymous versus signed-in comparisons.

Controls that held are reported too, in a "Controls that held" section. That way a later fix does not break a protection that worked.

> **Demonstration: a false positive caught by a control:** This is a demonstration on a fictional app, Harbor Notes, hosted on example.com. No real customer system was tested.
> 
> An agent sends a request to rename a note owned by another account:
> 
> ```http
> PATCH /api/notes/9001 HTTP/1.1
> Host: harbor.example.com
> Cookie: session=
> Content-Type: application/json
> 
> {"title": "renamed by B"}
> ```
> 
> The response is:
> 
> ```http
> HTTP/1.1 200 OK
> Content-Type: application/json
> 
> {"ok": true}
> ```
> 
> The agent first records "possible broken object-level authorization". The validation step reads the note back as account A, the owner:
> 
> ```http
> GET /api/notes/9001 HTTP/1.1
> Host: harbor.example.com
> Cookie: session=
> ```
> 
> ```json
> {"id": 9001, "title": "Quarterly plan", "owner": "a@example.com"}
> ```
> 
> The title did not change. The server returned `200` and ignored the write. A control with a nonexistent ID (`/api/notes/0`) also returned `200 {"ok": true}`, which shows that this endpoint answers success to anything. The finding is rejected. The report lists it under controls and notes, not under findings.
## What is the validation checklist for an AI-generated finding?

Use this checklist on every AI-generated finding, whether it comes from us or from another tool. A finding passes only if each box can be ticked from the saved evidence.

1. The request and response are saved, with a timestamp.
2. The test used accounts and records that the tester owns, marked as test data.
3. For a write or delete, a second session read the state back and shows the change.
4. At least one control ran beside the exploit and behaved as expected.
5. The data shown is not public by design.
6. The steps reproduce on a second attempt.
7. The report labels the evidence: "Observed" (reproduced), "Browser-observed" (shown in a real browser), or inferred.
8. A human reviewed the evidence before the finding reached the report.
9. The impact statement does not go beyond what the evidence shows.

If items 3, 4 or 8 are missing, treat the finding as unconfirmed. [Human-in-the-loop pentesting](/guides/human-in-the-loop-pentesting/) shows where review points sit in the workflow.

## How do you handle findings that cannot be proven?

Findings that cannot be proven go into a separate section, not into confirmed impact. Our reports use a "Limitations and open questions" section for these. An inferred path is labeled as inferred, so a reader can see what was proven and what was only suspected.

To check any AI pentest report, count how many findings carry a read-back or a control, and how many rest on a status code. The [guide to evaluating an AI pentest report](/guides/evaluating-an-ai-pentest-report/) goes through the signals.

## Where can you see this in practice?

The [example assessment](/example-assessment/) is a labeled demonstration with a rejected false positive beside a validated finding. [How AI pentesting works](/how-it-works/) shows where the evidence check and the human review sit in the flow. Excession Labs checks the evidence for each finding and writes the report.

## Questions

### Do human pentesters also report false positives?

Yes. Any tester can misread a response. The difference is scale and tone: an agent can produce many plausible-looking claims fast, each written with confidence. That makes a fixed validation step more important, not less.

### Is a false positive worse than a missed bug?

Both cost you. A false positive wastes engineering time and teaches developers to ignore reports. A missed bug leaves a hole open. A good process keeps both rates low and states what it did not test.

### How can I check a finding without running the exploit?

Ask for the saved request and response, the timestamps, and the read-back that shows the changed state. If the report has only a status code and a description, ask for more before you accept the finding.

### Can you remove false positives completely?

No. You can lower them with evidence rules, controls and review. A report should also list its open questions, so you can see what was inferred and not proven.

## Sources

- [OWASP API Security Top 10 2023: API1 Broken Object Level Authorization](https://owasp.org/API-Security/editions/2023/en/0xa1-broken-object-level-authorization/)
- [OWASP Web Security Testing Guide v4.2: WSTG-ATHZ-04 Testing for Insecure Direct Object References](https://wstg.owasp.org/v4.2/4-Web_Application_Security_Testing/05-Authorization_Testing/04-Testing_for_Insecure_Direct_Object_References/)
- [Excession Labs: Incursion](https://excessionlabs.ai/incursion/)