We Benchmarked Our Own Crawler on 131 Applications. It Found 61%.

Every security tool on the market publishes how many vulnerabilities it finds. Almost none publish how much of your application they actually reached. That second number decides the first one, because a tool cannot exploit an endpoint it never saw, and yet hardly anyone measures it. We built a way to measure it, we ran our own crawler through it, and the results are not flattering, so I am publishing them.

The short version: on 131 test applications that we designed ourselves, our crawler mapped 61.4% of the reachable API surface. On real open-source software that we did not write, the median was 13.3%. The gap between those two numbers is the most useful thing in this post, and I will explain exactly where it comes from.

Internal measurement, July 2026: These figures describe that run, not the crawler today. They have not been independently validated, because the raw run data and correction overlay are not published.

The question nobody asks

I have spent a large part of my career reading other people's penetration test reports, first as a consultant, then running teams, then sitting on the other side of the table reviewing vendors. The finding count is never the interesting part. Twelve findings, forty findings, it tells you almost nothing on its own. The interesting part is always what is not in the report at all, and the honest answer is that most of the time nobody knows, including the tester.

Many automated web security workflows start by discovering the application, which means crawling, following links, submitting forms, watching the network traffic, building a map of what exists. Testing then uses that map alongside supplied specifications and other inputs. A detection rate on its own says nothing about discovery coverage, so ask for both numbers and their denominators.

Why the question goes unanswered

You cannot score discovery without ground truth. To say a crawler found 61% of an application you first need an authoritative, complete list of what that application actually exposes, every route, every method, every parameter, and for real software that list almost never exists in a trustworthy form. Documentation drifts, OpenAPI specs are partial, and asking the tool itself what exists is circular reasoning.

So we built the ground truth. We have 131 containerised test applications, each one deliberately expressing the same domain through a different architecture, and each one shipping a generated ground_truth.json that enumerates its real surface. GraphQL with a single endpoint. Server-rendered Django. A React SPA that only talks JSON. Rails with Hotwire. An API with no HTML to crawl at all. Same problem, twenty different shapes, because the shape is exactly what breaks crawlers.

How we measured it, stated plainly

The metric that matters is what we call real coverage, and the definition is deliberately unkind to us:

real_coverage_pct = reachable_hit / reachable_canonical

Read the denominator carefully, because that is where benchmarks usually cheat. It is not everything the crawler captured, and it is not everything that exists. It is every endpoint in the ground truth that was reachable in that run, so an endpoint sitting behind a role we never authenticated into does not get counted against us, and an endpoint we hit by accident that was never in the ground truth does not get counted for us. A few rules follow from that, and each one exists because the obvious alternative flatters the tool:

• Never quote captured over canonical. That is the optimistic denominator, it runs several points higher, and we have quoted it by mistake internally before. Anything we publish uses the reachable denominator.
• Infrastructure failures score null, not zero. If a container is dead the crawler did not fail, the harness did, and scoring that as 0% quietly launders a broken test environment into a real result. This one matters more than it sounds, and I come back to it below.
• Every endpoint gets a label, not just hit or miss. Captured with a body, captured without a body, discovered but never exercised, intercepted, missing. A URL you saw but never sent a request to is not coverage.

The run I am quoting is from 20 July 2026, commit 18a67a2, 131 applications, 2 hours 12 minutes of wall clock. Across 24 stored runs on the same commit we measure a standard deviation of 1.14 percentage points. That is the run-to-run variation on that commit. Telling a real change from noise still needs comparable samples and a proper statistical test.

Our numbers, including the bad ones

On the 131 applications we built, weighted across every reachable endpoint:

The mean across applications is 69.8% and the median is 73.3%, both higher than the weighted figure, which tells you the applications with the largest surfaces are the ones we do worst on. That is the honest reading and it is not a comfortable one.

That leaves 697 reachable endpoints the crawler never reached. Missing a state-changing operation matters because it can leave an important workflow untested.

The architectures that beat us, with their full surface size:

Server-rendered frameworks and single-endpoint GraphQL are where discovery goes to die, and the reason is structural rather than a bug we can patch away in an afternoon. When the whole API is one POST to /graphql, a crawler that thinks in URLs has nothing to enumerate. When Rails Hotwire returns HTML fragments over the wire instead of JSON, there is no API-shaped traffic to observe at all. We know why we score zero there. We have not fixed it yet.

The number that actually matters

Everything above is measured against applications we designed, and you should discount it accordingly, which is precisely why we also ran against real software. We took 16 self-hosted open-source applications that ship their own OpenAPI specification, converted each specification into ground truth, and scored the crawler against software nobody on our team wrote.

Across the nine applications where administrator authentication worked correctly, the median real coverage was 13.3%. Weighted across all their endpoints together it was 14.5%, 370 endpoints out of 2,544.

NetBox is the one to sit with. It exposes 1,194 documented endpoints and our crawler reached two of them. Jellyfin exposes 388 and we reached sixteen. If you are running any automated security tool against software with a surface that size, and you have never asked the vendor what fraction of it the tool reaches, you now have a question worth asking.

I want to be fair about what causes this, because the honest explanation takes some of the drama out of it. A user interface crawler exercises the endpoints the interface calls, and it is being scored here against the application's entire documented REST surface, most of which no button in the product ever touches. NetBox's 1,194 endpoints are a full CRUD API over an enormous data model, and the web interface calls a small slice of it. So this is a recall floor by construction, and a spec-aware probe that reads the OpenAPI document directly would score far better.

That caveat cuts both ways though. Your attacker also does not restrict themselves to what the interface calls.

What this does not measure, before anyone tells me

I would rather state these myself than have them handed back to me later.

1. This measures discovery, not exploitation. Nothing here says anything about whether a vulnerability gets found once the endpoint is mapped. Our test applications are not deliberately vulnerable and carry no vulnerability ground truth, so this is not a pentest benchmark and I will not let it be described as one.
2. The synthetic corpus is variations on one domain. 131 applications sounds like a lot, and the architectures really are different, but they are 131 expressions of the same problem space. Architectural diversity is real, domain diversity is not.
3. The median uses nine authenticated applications, not all sixteen. Some had dead containers or authentication configuration that did not match how the application actually logs in, gitea needing a CSRF token in the form body being the clearest case. Those scored zero. I have deliberately excluded all of them from the 13.3% median rather than banking someone else's broken login as our own recall failure, because that is exactly the kind of quiet accounting that makes benchmarks worthless.
4. One crawler, one configuration, administrator role only. Single role, deterministic mode, shallow completion bound. A deeper run with multiple roles would score differently and I have not run it.
5. There is a hand-authored correction overlay. When the generated ground truth is wrong, and it sometimes is, we correct it, and every correction moves a denominator. That is the single most attackable thing here and I am not going to pretend otherwise. Any public release ships those corrections as reviewable files with per-entry evidence, frozen before anyone is scored, or it is not worth releasing.
6. We cannot yet report request-body capture. Knowing an endpoint exists is not the same as knowing the shape it accepts, and a write operation without a body shape cannot be fuzzed downstream. That field is not populated in this run, so I have no number for you, and I am not going to estimate one.

What we are doing with it

A reproducible public release needs the corpus, ground truth, scorer, correction overlay and input adapters for other tools. None of that is downloadable yet and there is no release date, so treat these figures as internal measurements until someone else can run them.

One design decision worth stating in advance: we will not be running competitors ourselves and publishing their scores. Plenty of vendor licences prohibit exactly that, and grading your rivals with your own ruler is not credible even when it is legal. Any release will carry the ruler and our own numbers, and results for anyone else's tool arrive by pull request with the artefacts attached.

I am aware of how this reads. Publishing a benchmark makes us responsible for its fairness, and if it is ever shown to be shaped around our own product then the damage is worse than anything this post earns. That is the trade, and we are taking it, because I would rather argue about a number in public than keep pretending the industry has one.

61.4% on applications we built. 13.3% on applications we did not. If your tooling does better, show me the artefacts.