Where AI pentesting falls short: limits you should know before you buy

AI pentesting has real limits. It does not know your business rules unless you write them down. It can miss a novel multi-step chain. It cannot test people or buildings. For each limit below, we say how Excession Labs handles it or that it is out of scope. A buyer who knows the limits before requesting a test reads the report better.

What are the main limits of AI pentesting?#

AI pentesting is strong on repeatable, evidence-driven checks of web apps and APIs. It is weaker where the answer depends on context that is not in the traffic. The table lists six limits and our position on each.

Why is business context a limit?#

Business context is what makes a response correct or wrong for your product. A scanner or agent can see that a request returned 200. It cannot know that a refund of a negative amount breaks your finance rules unless someone says so.

Excession Labs handles this at scoping. We agree the scope in writing with you: assets, roles, test accounts and the actions you care about. Name your risky workflows, such as invitations, billing changes or exports, and we test them with evidence.

A gap remains where we were not given a rule. The report states it under "Limitations and open questions", and it appears neither as a confirmed finding nor as a clean result.

Can AI find novel multi-step attack chains?#

Sometimes, but do not rely on it. A novel chain joins several small weaknesses in a way nobody has listed before. Human specialists with deep product knowledge often find these.

Our evidence standard limits the damage of a wrong guess. A finding needs evidence of a real effect: a write is proven by reading the changed record back through another session, and a bare "success" response is not proof. If a path is plausible but not reproduced, the report labels it as inferred, apart from the confirmed items.

We do not claim that Incursion finds all vulnerabilities. If your risk is a high-value target with unusual logic, raise it on the call. The AI vs manual pentesting page explains when human specialist work fits better.

What if the authorization rules are ambiguous?#

Authorization testing needs a rule to test against. "Users can only see their own invoices" is a rule. "Managers see what they need" is not. The tool cannot invent the missing part.

The method uses differential tests. Incursion compares anonymous and signed-in callers and reads back changes from a second account. Controls run next to each exploit request: nonexistent identifiers, wrong record types, another account's token. How AI agents test authorization has a worked example.

When two behaviors both look plausible, the report lists the question and you decide which behavior is intended.

How does AI testing treat fragile or rate-limited systems?#

Fragile systems can fail under normal test load. Rate-limited systems can block the tester, or charge you for each call. OWASP lists unrestricted resource consumption as a top API risk, so a rate limit is also something to test, with care.

Each tool in Incursion has a rule: it runs on its own (allow), waits for approval (ask), or is blocked (deny). The operator sets the rules. Tests of uploads, password guessing, email sending and paid third-party APIs are bounded, and their cost is stated in the report.

If a system is very fragile, test a staging copy. The production safety guide has the stop conditions and a safety checklist.

What can AI pentesting not do at all?#

It cannot do social engineering or physical access tests. Phishing, phone pretexts, badge cloning and tailgating involve people and places, and web traffic does not show them.

These tests are out of scope for aipentestservices.com, as are internal network tests and other work outside web apps and APIs. NIST SP 800-115 (published in 2008, still listed as final) treats social engineering as a separate technique from technical testing. Pick a provider for those tests on its own merits.

Does model nondeterminism change the results?#

Yes. A language model may take a different route on a second run, so two runs of the same scope may not find the same set of issues.

A confirmed finding stays reliable, because each finding links to its evidence, with saved requests and responses and timestamps. You can replay them. Key evidence can carry a SHA-256 manifest, so you can check that the files did not change after the test.

One clean run is not proof that nothing is there. Add coverage over time: test after meaningful releases and retest fixes. See per-release testing.

What should you ask before buying an AI-assisted test?#

Ask these five questions of any provider, including us:

1. Which of my business rules did I give the tester, and where are they written?
2. What would a human specialist add for my riskiest workflow?
3. Which tools may run without approval, and which are blocked?
4. Where does the report list what was not tested?
5. Can I replay a finding from saved evidence?

A provider that cannot answer these has told you something. To check a finished report, see how to evaluate an AI pentest report.