Why MCP Checkup doesn't run your code (and what that means for stdio servers)
Today MCP Checkup lists remote MCP servers only: endpoints with an HTTP URL that any client can connect to. Maintainers of stdio servers ask us, reasonably, when their turn comes. The answer needs a short detour through what a stdio server actually is.
An address versus a package
A remote server exists whether or not anyone is looking at it. It has a URL, it answers requests, and observing it means sending it the same requests a client would. Nothing about that requires trusting the server’s code; we are on the outside, reading what comes back.
A stdio server has no URL. It is a package on npm or PyPI. It only exists as a process when someone downloads it, installs it (and npm install already executes arbitrary code via lifecycle scripts), and launches it as a child process of their own client. There is no “probing” a stdio server. There is only the question: on whose machine does this code run?
That question has three honest answers, and they are not equally trustworthy.
The line we hold today
We do not execute maintainers’ code on our infrastructure. Not in a container, not in a sandbox, not “just to get tools/list”. The reason is not squeamishness. A checker that runs untrusted packages is a machine that installs whatever the internet hands it, on a schedule, with credentials to sign the results. Getting that right is a serious security project with its own review, and until that review is done and published, the correct amount of code execution is zero.
This is also why, on remote servers, the probe never calls one of your tools. tools/list describes what a tool would do; calling a listed tool does it. We stop at the description. The one tools/call the probe does send names a tool that does not exist, __mcpcheckup_probe_nonexistent_tool__, only to see how the server refuses it.
The three tiers we are designing toward
None of the tiers below is live yet. This is the design, published so maintainers of stdio servers can see where things are going and tell us if it is wrong.
Tier A: static analysis, no execution. Fetch the pinned tarball, do not install it, do not run it. Read package.json for the declared SDK version, entry point and engine constraints; parse the source for tool definitions that are registered statically; note whether lifecycle scripts exist; check dependencies against known vulnerability data. Everything this produces is marked as statically analysed. Anything it cannot determine, such as tools registered at runtime or the real tools/list, is marked not checked, with that reason.
Tier B: your CI runs the check. A GitHub Action in your own repository fetches, installs, launches, performs the handshake and tools/list, and computes fingerprints, on your runner. Only fingerprints and metadata come back to us; no source, no artefacts. The results are marked as self-attested, because that is what they are: your infrastructure saying what it saw.
Tier C: independent execution in isolation. Once a security review of hosted execution is complete, and only with the maintainer’s authorisation, we run the same checks in an isolated environment. Those results would be marked independently observed, and they are the only tier that could honestly carry the same weight as a remote probe.
The three tiers carry different trust, and the design requires that a report page and the API say which tier produced each assertion, in the field and on the screen. “Your CI said it passed” and “an independent party reproduced it in isolation and signed the result” are different products, and collapsing them into one badge would be a lie of omission.
What to do meanwhile
If your server has a remote mode, that endpoint can be listed and claimed today, and everything on this site applies to it. If it is stdio-only, there is nothing to do yet, and we would rather say so than list a page that shows a package name and fifteen rows of “not checked”.
When tier A ships, we will say so here, and its limits will be written on every page it produces.