Why we don't score MCP servers
People ask for a number. “Just tell me if this MCP server is OK.” We understand the request and we are not going to satisfy it. This post is the long form of why.
What a check actually is
Every check we run answers one narrow question with one of four outcomes:
- Verified. The property we looked for was there, with evidence.
- Failed. The property was absent or wrong, with evidence.
- Observed risk. Something is worth a human’s attention, but it does not let us conclude anything about the server as a whole.
- Not checked. We could not test this, and the row says why: for example no coverage, no authorisation, a probe that ran out of budget, or testing it would require running code we won’t run.
Nothing above is a percentage. A server with twelve verified rows and one failed row is not “92%”. The one that failed might be a broken handshake; a row marked observed risk might be a tool description that instructs the model to ignore its system prompt. Those do not weigh the same, and no fixed weighting we could publish would be right for both a hobby weather server and a payment API.
The number would become the product
The moment a summary number exists, three things happen, all of them bad.
It becomes the thing people optimise. Maintainers would tune for the metric rather than for the property behind it. That is the same dynamic that made SEO scores worthless as a measure of content.
It becomes something we could be paid to move. We charge maintainers for faster checks, a live badge, faster alerts and room for more claimed servers. If a score existed, every pricing conversation would carry the implicit question of whether paying affects it. We would rather make that question impossible to ask. The rule on our side is simple: conclusions are not for sale, and the cleanest way to enforce it is to have no conclusion to sell.
It would be read as a decision we didn’t make. A green grade says “go ahead”. We did not look at your data handling, your backend, or what your tools do when called. We looked at the protocol surface from the outside. A verdict would claim more than the evidence supports.
What we do instead
We show every row with the kind of evidence behind it and, on a signed run, its state; the page says when that run happened. The visual language is deliberately flat: four states, each with a shape, a word and a colour, so the page reads the same in a grayscale screenshot. Unsigned runs use the same layout without colour and without a state on any row: each row shows what the check tests and any reason we recorded, but no verdict, because no signature stands behind them. A page is never made to look worse to nudge someone into paying.
Each run that meets our publication rules gets a signed attestation, claimed or not: a statement that on this date, our probe observed exactly these things, which anyone can verify without asking us, using the public keys we publish. Note what the signature covers, and what it doesn’t. It covers that the observations are ours and unaltered. It does not say the server is good. Unsigned runs are shown without colour and labelled as unsigned for the same reason: we never let an observation dress up as a verified conclusion.
The same rule holds for the README badge. On the free plan it says “claimed” and nothing else. On the paid plan it shows the worst conclusion among the checks that actually ran in the most recent run, only when that run is signed; an unsigned run shows no result at all.
“But I still need to decide”
You do, and the page is built to help you decide rather than to decide for you. Read the failed rows first; each says what we observed, and you can reproduce it with curl. Read the observed-risk rows next and judge them against what your agent will actually let the server do. Treat “not checked” as exactly that: absence of information, not a quiet pass.
If that sounds like more work than reading a number: it is. It is also the only honest amount of work, and it is the same work we would have had to hide inside a number to give you one.