We asked 1,144 MCP servers for a made-up tool
The one tools/call a probe sends names a made-up tool, __mcpcheckup_probe_nonexistent_tool__, and the check called error_taxonomy looks at how the server refuses it. A well-formed JSON-RPC error, or an HTTP 401 or 403, and the check holds. Any other answer, and the row is recorded as a risk signal.
Until this week, that was all the row said. On 2026-09-20, of 579 servers with a signed result, 338 had that row marked as a risk, and every one of them carried the same sentence: the error scenario did not return a protocol-shaped JSON-RPC error. Fifty-eight percent of the signed servers, and neither the maintainers nor we could tell from the page what those servers had actually answered.
What the specification says, and what servers do
The MCP specification’s tools chapter, in every revision from 2025-06-18 through the current 2026-07-28, draws one line. Protocol errors, which include an unknown tool, are standard JSON-RPC errors; the example given is {"code": -32602, "message": "Unknown tool: invalid_tool_name"}. Tool execution errors, such as an API failure inside a tool that does exist, are reported inside a normal result with isError: true. The 2025-11-25 revision moved input validation from the first group to the second; the unknown tool stayed in the first.
Many servers do the opposite for an unknown tool: they return a result, isError: true, and a text block that says the tool was not found. We know this pattern well, because our own MCP server did it. On 2026-09-14 a user’s agent called a nonexistent tool on mcpcheckup.com/mcp and got HTTP 200 with a result whose text read “MCP error -32602: Tool … not found”, wrapped in isError: true. Our own checker had been recording our own server as a risk for the same reason. The fix was a small dispatcher in front of the SDK: a single tools/call whose name is not in the registered set now gets a JSON-RPC error with code -32602 before the SDK sees it. Everything else, including argument validation, still goes through the SDK and still comes back as isError where the specification says it should.
What we changed in the check
We did not change the verdict. The same input still yields the same held-or-risk outcome as before; a differential test over 9,828 constructed inputs and a reviewer’s 450,000 random ones found zero cases where the new code and the old code disagree. What changed is what the risk row records, in a bounded form:
- the HTTP status the server answered with;
- the media type of the answer, from a fixed list of four, or “none” or “other”;
- one of seven categories: a result marked
isError, a result not marked as an error, a self-declared JSON-RPC message with a malformed error, valid JSON that is not JSON-RPC, a body that is not JSON, an empty body, or an event stream with no data event that carries anything; - for the malformed-error category only, the JSON-RPC error code, if it is an integer small enough to be stored exactly.
No text from the server is stored. A category, a status code, a media type from a list, an integer. The report page and the MCP tools print a sentence for each category, so a maintainer reading “The server answered with an ordinary result marked isError instead of a JSON-RPC error object” knows which line of their handler to look at. The check suite moved from 0.4.0 to 0.5.0, and the signed results say which version produced them.
What the first day shows
The new probe went live on 2026-09-23 at 01:08 UTC, and suite 0.7.0 replaced it at 16:47 UTC the same day (0.6.0, deployed in between, did not change this check). Taking each server’s most recent run in that window, this is what the error_taxonomy row recorded (1,156 servers had a run in it; the other 143 of the 1,299 in the directory did not, because the probe ran no cycles between 11:45 and 16:50 UTC that day):
| outcome | servers |
|---|---|
| held (JSON-RPC error, or a credential gate) | 743 |
risk: result marked isError |
359 |
| risk: body is not JSON | 14 |
| risk: valid JSON, not JSON-RPC | 13 |
| risk: empty body | 8 |
| risk: result not marked as an error | 7 |
| risk: malformed JSON-RPC error | 0 |
| risk: event stream with no data | 0 |
| not checked (probe stopped earlier, rate limited, or out of budget) | 12 |
Of the 401 servers recorded as a risk, 359 answered with a result marked isError: nine in ten. Of those, 285 answered as plain JSON and 74 as a server-sent event stream carrying the same shape. The remaining tenth, 42 servers, is a mixed bag: 7 answered with an ordinary result not marked as an error, and most of the other 35 do not look like an MCP server at that path at all: 404 pages, redirects with empty bodies, three 402s, two 451s, four 5xx.
That distribution is the point of the change. Before it, one sentence covered a server whose only fault is the SDK’s default error shape and a server that answers tools/call with an HTML 404. Those are not the same finding, and a maintainer deserves to know which one they have.
Suite 0.7.0 draws that line one step earlier: it does not send the made-up call when it could not read a complete tool list for a reason other than a credential gate, or when that list names the made-up tool, because then it cannot rule out that the server really has a tool by that name. By 01:08 UTC on 2026-09-24 every one of the 1,299 servers had a run under that version, and the same row then read: 802 held, 401 answered with a result marked isError (312 as plain JSON, 89 as an event stream), 8 answered with a result not marked as an error, 3 answered 402 with JSON that was not JSON-RPC, 73 were not sent the call because the tool list was paged or could not be read, and 12 were not checked for other reasons. The 404 pages and empty redirects from the first table are now among those 73: they did not return a readable tool list either, so the call is no longer sent, and the row records that we could not see the full list instead of how the server answered.
Update, 2026-10-08: since suite 0.8.0, released on 2026-09-26, the probe also holds the call back when the handshake or the tool list is behind a credential gate, and records the row as Not checked. The sentence above describes 0.7.0 only. The call we hold back covers the current rules.
What we are not doing yet
We are not moving the isError case out of the risk column. The specification is clear that an unknown tool is a protocol error, the row is factually correct, and changing what counts as a risk is a decision about the check’s meaning, not about its evidence. It also changes every signed result going forward, so it goes through the same review as any other change to a check. If we ever change that, the change will be visible in the check suite version, not made quietly.
What we would say to a maintainer today: if your server is in the 359, it is almost certainly one line. If your SDK lets you return a JSON-RPC error for an unknown tool name, that is the change; if it does not, a dispatcher in front of it, like the one we added to our own server, is a few lines and puts you among the servers that hold.
Where the numbers come from
The 338-of-579 baseline is from a read-only query against the production database on 2026-09-20, counting signed directory rows whose latest error_taxonomy assertion was a risk. The table above is one read-only query over the same database: for every target, its most recent non-trial run started between 2026-09-23 01:08 UTC and 16:47 UTC, grouped by that run’s error_taxonomy outcome and, for a risk, by the recorded reason, HTTP status and media type. The 0.7.0 figures are the same query with the window moved to 01:08 UTC on 2026-09-23 through 01:08 UTC on 2026-09-24. The 9,828 and 450,000 figures are the differential test in the check suite’s repository and the reviewer’s fuzz run recorded in that pull request. The specification quotations are from the two linked revisions of the tools chapter.