What a Pentest Requires, Part 2: Methodology, Depth and Evidence

A finding count on its own tells you nothing. Forty findings from a tool that only looked at your login page is worse than eight from someone who worked through the whole application. What makes the number mean something is the three things underneath it: a stated methodology, depth of testing, and evidence you can reproduce without the tester in the room.

This is part two of a series on what a penetration test actually requires. Part one covered licensing and scope.

Methodology: what was tested, and what was not

A real penetration test follows a named, written methodology, stated up front, so that you and your auditor can see what was covered. Examples include the OWASP Web Security Testing Guide (v4.2 is the stable edition) and PTES. The OWASP Top 10 is a useful risk-awareness reference, rather than a complete test methodology. Naming a reference is not decoration. It helps turn "we tested your application" into a claim someone can check.

A methodology section that does its job tells you four things:

• Which standard was followed, by name and version.
• Whether both the unauthenticated and the authenticated surface were tested. An unauthenticated-only test misses most of an application, and plenty of reports quietly are one.
• Whether each issue was validated by hand or taken from a tool's output without a second look.
• What was covered, counted as executed test cases by category and role, with exclusions listed and planned checks kept apart from the ones actually completed.

That context is what makes the finding count mean something. Eight findings alone do not tell you whether the team worked through the important roles and workflows or stopped at the first page. If the methodology or coverage is missing, ask the provider for it before you rely on the report. A larger count is not proof of better coverage.

Depth: a proven finding versus an observation

Methodology tells you what was tested. Depth tells you how far each test was taken, and it is the clearest line between a scan and a pentest.

A scanner flags a missing security header and stops. A tester treats that as a signal and follows it: what does this header protect, is that protection actually needed here, what else is weak on this endpoint, and can any of it be chained into something an attacker would care about. The output of the first is a list of hygiene items. The output of the second is a proven attacker result.

Depth is also where chaining lives, and chaining is where the serious results come from. An information leak on one endpoint is a Low. A weak password policy is a Low. A leaked password hash plus an offline crack plus an administrator account with no step-up authentication is an account takeover. A check that evaluates each signal independently may miss the connection.

On one recent engagement the chain ran like this: a password hash exposed through an unauthenticated endpoint, cracked offline in about a second because the password was weak, giving platform administrator access. From there, the ability to read and modify other customers' data. Every individual link was a finding another tool could have found. The chain is what made it a breach, and the chain is what the assessment needed to demonstrate.

Evidence: could someone else reproduce this?

A finding is only real if someone who was not in the room can reproduce it. That is the whole standard, and it is stricter than it sounds.

A finding that meets it gives the developer, at minimum:

• The exact request that triggered it, in full, including headers and body.
• The response that proves it, showing the actual data returned or the actual state change, not a description of it.
• Reproduction steps a developer can follow on their own environment without guessing what was meant.

A pattern I have seen in report reviews is a paragraph asserting that an issue exists, a severity label, and a generic remediation. No request, no response, nothing the developer can run. That puts the burden of proof on the person who has to fix it, which is exactly backwards, and it is why so many findings get disputed into oblivion and quietly closed.

There is a practical reason to care beyond principle. When you retest, you need to demonstrate the issue is gone. If the original finding never contained a reproducible request, nobody can prove the fix worked, and you pay for that ambiguity twice.

The questions to ask

1. Which methodology, by name and version? If the answer is "industry best practice", that is not a methodology.
2. How many test cases were executed, in how many categories? This is the number that makes the finding count mean something.
3. Was the authenticated surface tested, and under which roles? Most of an application lives behind the login.
4. Show me one finding end to end. Pick a Critical at random and ask for the request, the response and the steps. One example tells you what the other thirty look like.

Part three covers severity scoring and what a report has to contain to survive an audit.