What a Pentest Requires, Part 4: Quality, Remediation and Honesty

The last three requirements are the ones that decide whether a report helps you or costs you money. Is every issue in it true and listed once? Does the fix work on the stack you actually run? And is the vendor honest about whether a machine or a person did the work? Each of these fails quietly, and each one is expensive in a way that only shows up weeks later.

This is the final part of a series on what a penetration test actually requires. Earlier parts covered licensing and scope, methodology, depth and evidence, and severity and reporting.

Every issue once

An inflated finding list is not a harmless presentation choice. A small engineering team reading forty tickets believes it has forty problems, and spends its limited time working through one misconfiguration filed under four different titles as though each were a separate emergency. Meanwhile a longer list makes a shallow scan look like a substantial piece of work, which is usually why it happened.

In a report I went through recently, 15 of the findings were duplicates of just four real issues:

Whether a tool or a person proposes the grouping, someone has to read every finding, recognise that nine tickets share one root cause, and merge them. That is a human editorial pass, and its absence is visible in the output every time.

Every issue true

Missed vulnerabilities are the obvious risk. False positives do their own damage. A false positive is a client acting on something that was never real, spending time and money on it, and losing confidence in the rest of the report when they find out.

The worst version I have seen recently was a reported cross-user data exposure, written up with a data protection impact attached. It did not reproduce. On a live retest, all 45 of 45 records returned belonged to the requesting account. The access control worked correctly and there had never been a breach. But a client reading that report in good faith might notify a regulator about an incident that did not happen, and unwinding that is far more expensive than the test cost.

A related tell worth knowing: check whether any "recently disclosed CVE" section is real. One report claimed to have tested vulnerabilities disclosed in the trailing seven days before its May 2026 publication date, then listed CVE identifiers from 2023 and 2025. That section was boilerplate, and once you find one section that is boilerplate you should assume there are others.

Remediation that works on your stack

Finding the problem is half of what you pay for. The other half is a fix written for the actual cloud, framework, gateway and database you run, rather than a generic template. A wrong fix is worse than no fix, because the client applies it, closes the ticket, and believes they are safe while the hole is still open.

A clean example: a vendor found a genuine server-side request forgery, a real and serious issue, and told the client to enable IMDSv2. IMDSv2 is an Amazon Web Services feature. The target did not run on AWS. It ran across other cloud providers, and the advice needed to name the metadata protection those providers actually offer. Google Cloud, for example, protects its metadata service differently, by requiring a specific request header. So the single remediation offered for the most serious issue they found would have done nothing at all on the environment they had just spent the entire engagement testing.

That is a review failure. Whatever the testing process, the recommendation did not match the infrastructure it was supposed to help protect.

Honesty about how the work was done

You are entitled to a straight answer about whether a machine or a person did the testing, because it tells you what you are buying and who stands behind the result.

Ask what a vendor means by "fully automated": test execution, evidence analysis, report drafting or the entire engagement. A human-written summary does not establish who ran the tests. Ask separately who reviews the findings, what they verify and who is accountable when the output is wrong.

Our own model is deliberately explicit, and I would rather state it than let anyone infer it. The AI runs the tests from ManticoreAI's own attack infrastructure. Inference uses Microsoft-hosted OpenAI models through Azure Foundry. Microsoft processes prompts and outputs; they are not made available to OpenAI. A named consultant reviews and approves each Assess finding before it reaches you. We call that Auditor, not Author. It is why a human name appears on the report, and it is the part that makes the output defensible in an audit.

One more thing a retainer needs

Worth adding because it is invisible until you feel it: some engagements need a stable test version or a temporary deployment freeze. The testing window is a freeze window for the business. In Singapore anything beyond about five days of freeze is already considered long, and I have seen a small application take eleven days to test. For a single curiosity-driven assessment that might not matter. Across a portfolio of applications tested in parallel, it is the whole economics of the arrangement.

Speed here is not a marketing claim about being fast. It is a constraint on how much testing your business can absorb in a year.

The questions to ask

1. How many distinct root causes are in this report? Not how many findings. If nine tickets share one cause, you have one problem, not nine.
2. Pick a finding and ask them to reproduce it live. False positives do not survive this.
3. Does this remediation reference my actual stack? Cloud provider, framework, gateway, database. Generic advice means nobody looked.
4. Who did the testing, and who approved the findings? Get it in writing. If the answer is fully automated, ask what the review step is. If the answer is people, ask how that scales to your volume.
5. How long is my deployment freeze? Multiply it by the number of applications you need tested this year.

Across the four parts of this series that is twelve things a penetration test has to get right. Most of them are checkable before you sign anything, and the ones that are not become checkable the moment the first report lands. Measure whoever you use against them, including us.