PentestGPT: what it is, what the paper measured, and what it is not

PentestGPT is an open-source research tool that uses large language models (LLMs) to guide a penetration test, and since version 1.0 to run one as an agent. It comes from a USENIX Security 2024 paper, where it solved easy and medium lab machines and failed on hard ones.

This page covers what the paper measured, what changed since, and what to check before you rely on a PentestGPT result. For the wider picture of models in testing, see LLM for penetration testing.

What is PentestGPT?#

PentestGPT is a Python project in the GitHub repository GreyDGL/PentestGPT, under the MIT licence. The repository says the tool is "for educational purposes and authorized security testing only".

The original design, now kept as pentestgpt-legacy, has three cooperating LLM sessions: one plans, one turns the plan into commands, and one reads tool output. A person runs the commands and pastes the results back. The paper calls this a human-in-the-loop strategy.

Version 1.0, which the README calls the "Agentic Upgrade", runs tools by itself through a commercial coding agent. It has a CTF mode (the default) and a pentest mode with the stages asset discovery, vulnerability identification and report.

Several other websites use the name. The paper and the project site link to the GitHub repository, so start there.

What did the PentestGPT paper measure?#

The authors built a benchmark of 13 targets from HackTheBox and VulnHub, split into 182 sub-tasks across 26 categories and rated easy, medium or hard. They told the tool to use only non-automated testing tools.

Two more results come from outside the benchmark. On 10 active HackTheBox machines, PentestGPT solved 4, at a total OpenAI API cost of 131.5 US dollars. In the picoMini CTF it scored 1,500 of 4,200 points and placed 24th of 248 teams.

The headline number on the project site, a 228.6% gain, is the increase in sub-task completion over GPT-3.5 used directly. Against GPT-4 used directly, the paper reports a 58.6% gain. Both are relative gains on the authors' own benchmark.

Where does PentestGPT fail?#

The paper lists three failure types from its own runs:

1. Images. The models could not read images that some targets needed.
2. Context-based guesses. It could pull names from a web page but did not build a wordlist from them, which a human tester would do.
3. Exploit code. It struggled to write working exploits within a limited number of tries, especially with low-level bytecode.

The authors also saw it try to brute-force SSH before running a vulnerability scan. On hard targets they wrote that the design "does not expand the LLMs' knowledge of vulnerabilities". They also note that hallucination is not fully prevented, which is why they kept a person in the loop.

These results are for the 2023 and 2024 models in the paper. Newer models may do better. No one has published a repeat of the 13-target benchmark with them that we could find.

What does the 86.5% result on the README mean?#

The README reports an 86.5% success rate (90 of 104) on the XBOW validation benchmarks, from an experiment in December 2025. The README itself calls it "a historical research result" and says it does not promise the same score from the current agent.

Read it with the benchmark's own warning. The XBOW repository now says that, as of mid-2026, the set is "saturated" and its vulnerabilities "have been trained into the models", and it is kept for historical purposes. These are 104 web CTF challenges with one hidden flag each. A high score shows a tool can find planted flags in small apps. It does not show how the tool does on your app, with your roles and your business rules.

Is PentestGPT a pentest you can hand to a customer or auditor?#

On its own, no. It is a research tool, and the project does not offer what a commissioned test needs. Use this table to see what you would have to supply yourself.

If you run it, do it on a staging copy, with written authorization, and treat every output as a lead until you reproduce it. The false-positives guide has checks that work on any tool's output.

How is PentestGPT different from an AI-assisted pentest service?#

The difference is who carries the responsibility. With PentestGPT, you set the scope, choose the model, watch for unsafe actions, check each finding and write the report. With a service, a provider agrees the scope with you in writing and answers for the evidence.

Our own service uses AI testing agents for the repetitive work, and Excession Labs checks the evidence for each finding and writes the report. The AI pentest tools guide sorts tools and services into four types by who runs them and who checks the results.

To judge any report, whether it came from PentestGPT, a commercial tool or a service, use the 12-row scorecard for AI pentest reports. To see the format we deliver, read the sample report on excessionlabs.ai, the main site of Excession Labs, which also runs this site.

Questions to ask before you use PentestGPT output#

• Which version produced it: the legacy assistant or the 1.0 agent?
• Which model and provider ran it, and where did the prompts and tool output go?
• Was the target in scope and covered by written authorization?
• Can you reproduce each finding from the saved commands and responses?
• Which parts of the app did it not reach, such as signed-in areas or second roles?