We Won a Four-Way Bake-Off. The Scoreboard Was Wrong in Our Favour.

A channel partner ran a four-way evaluation of AI penetration testing tools against a deliberately vulnerable banking application. We came first by a wide margin. Then I went back through the scoresheet properly, and the margin was not real. It was an artefact of how the table was built, and it was built in our favour. This post is the corrected version, because a benchmark you cannot defend is worth less than no benchmark at all.

A note on the figures: these are my numbers from that evaluation, not a current ranking of products. The partner’s raw scoresheet is not public, so you cannot reproduce the comparison yourself.

The scoreboard as it was reported

The partner tested four tools against the same target and marked which of them found each vulnerability. Two were commercial AI pentest products, one was a commercial automated scanner, and one was an open-source scanner. I am not naming them, because the point of this post is the method rather than the leaderboard. After I added the findings our platform reported that were not on the original sheet, the table had 39 rows and looked like this:

82 against 18 is the kind of number a sales team frames and puts on a slide. I sent it to the partner at the time. I would not send it now, and here is why.

What is wrong with it

The partner's original sheet had 19 rows. Those 19 were what every tool in the evaluation was actually pointed at and scored against. I then added 20 more rows, the findings our platform reported that nobody else's sheet contained, and I did it in good faith, because leaving them out understates what the product found.

But look at what that does to the arithmetic. Of the 20 rows I added, the number where any competing tool was marked as detecting anything is zero. Not a low number. Zero. Those tools were never run against those 20 items, never evaluated on them, and could not have scored on them. So the denominator grew from 19 to 39 for us, and stayed at 19 for everyone else, while their numerator stayed frozen.

The corrected numbers

Score every tool against the same 19 rows, which is the only set all four were evaluated on, and the table changes shape:

We still win, and I still think we are the better tool on this target. But the honest gap over the nearest competitor is 26 points, not 64. Our own score drops from 82% to 63%. Everyone else's roughly doubles. If you are evaluating tools and someone shows you a chart like the first one, the question to ask is not "how did you score this," it is "how many rows did each tool actually get marked against."

Where our own 12 come from

One more level down, because the same problem can hide inside a corrected number. Our 12 out of 19 is not 12 marks made by the partner. It breaks down like this:

Two of the twelve are rows where the partner had not credited us and I said the detection was there. I still believe that, and we can produce the finding for both. But you should not have to take my word for it, so here is the number without them: 10 out of 19, or 53%. That is the floor, counting only what an independent party marked without any input from us.

So the defensible range is 53% to 63%, against 37% for the nearest competitor. The ranking does not change at either end, which is the only reason I am comfortable publishing any of it. If our self-corrections had been what put us in front, the right thing would have been to say the evaluation did not settle the question.

Three places I scored us down

Three of the judgement calls in that sheet cost us points, and I want to be explicit that I made them deliberately, because they are the same calls a careful assessor makes.

1. A SQL injection we found and then suppressed. Our platform detected the injection on the user-search function, then its own validation stage flagged it as a false positive and filtered it out. It never reached the report. I did not count it as a detection, because a finding the customer never sees is not a finding. It is a miss, and it is our miss.
2. Two cross-site scripting entries I refused to count. The sheet listed reflected XSS in the error message produced by a SQL injection. We grouped those rows with the injection and did not count them. Grouping like that needs evidence, because SQL injection and XSS can have independent causes and fixes: parameterized queries do not replace the context-sensitive output encoding needed to prevent XSS.
3. A cross-site request forgery on the login form. Detected, then suppressed as unproven, so it was not counted. Login CSRF still matters in general: it can force a victim into an attacker-controlled account. OWASP describes login CSRF and its mitigations.

Notice that all three of those make our number worse and the comparison more honest. That is the trade, and it is the right one. A tool that files every symptom as its own finding will always beat a tool that groups them, on any scoreboard that counts rows.

Severity is the other half of the problem

The sheet also carried an "expected severity" for each item, and our ratings disagreed with it repeatedly. That disagreement is not an error, it is the method.

A default severity looked up from the vulnerability class and a rating based on what was proven can be far apart. The injection on the registration form is a good example: it is error-based disclosure that leaks schema detail through verbose errors, and it does not give up the database. We rated it Low. On the same sheet another tool rated it Critical because the string "SQLi" appeared.

If you accept the lookup-table severity, that tool looks more alarming and therefore more thorough. If you want a risk register you can act on, you want the rating that reflects what was proven. You cannot have both, and a benchmark that scores on class rather than confirmed impact quietly rewards the wrong one.

The target was too easy to settle anything

One more limitation, and it is the one I raised with the partner at the time. The test application was a small deliberately-vulnerable banking demo: injection, cross-site scripting, request forgery, directory listing. That is a fine target for comparing automated scanners against each other. It is not a pentest.

Nothing in it has a payment checkout flow, a CAPTCHA, an OAuth handshake, multi-factor authentication, several roles with different permissions, or a business workflow that only breaks on the fourth step. Those are where real applications fail, and they are what separates a tool that reasons through an application from one that fires payloads at parameters. A bake-off on a simple target compresses the field and makes everyone look closer together than they are, in both directions.

What to ask when someone shows you a bake-off

I have been on both sides of these evaluations for a long time, first running them as a consultant and now being scored in them. Four questions get you most of the way:

1. How many items was each tool scored against? If the answer is not the same number for everyone, the percentages are not comparable. This is the one that broke the sheet above.
2. Who wrote the ground truth, and when? If a vendor added rows after seeing the results, including us, ask which rows and what happened to the denominator.
3. Are symptoms counted separately from causes? One root cause filed as four findings beats careful grouping on any row count, and it is worse for the developer who has to fix it.
4. Is severity from a lookup table or from confirmed impact? Ask for the evidence behind one Critical, picked at random. The answer tells you what kind of tool you are looking at.

We published our crawler's discovery numbers for the same reason, and they are not flattering either. On test applications we designed, it mapped 61.4% of the reachable API surface; on real open-source software we did not write, the median was 13.3%. Measuring yourself honestly is uncomfortable and it is the only thing that makes the measurement worth anything.

We came first in this evaluation, and I would rather report the win I can defend, 53% to 63% against 37%, than the 82% against 18% I cannot.