User-agent: *
Allow: /
Disallow: /auth/
Disallow: /account
Disallow: /zh/auth/
Disallow: /zh/account
# Content-Signal (content-signal.org draft) — search=yes states plainly
# what the Allow line above already implies for every crawler; ai-input=yes
# covers a live/on-demand AI request reading this page to answer a question
# right now (e.g. an agent fetching a report page mid-conversation), which is
# different from TRAINING on it. No ai-train value is stated here: this
# site's existing policy already differs BY PATH (the eight named crawler
# groups below disallow training bots on result surfaces, functional pages
# and static assets), and
# Content-Signal carries no path granularity — writing ai-train=no here would
# contradict the training crawlers' own implicit allow on marketing pages
# just below (none of those groups have an Allow line there; the pages fall
# through unlisted), and ai-train=yes would contradict their Disallow on
# result surfaces. Silence on that one value, not a guess either way.
Content-Signal: search=yes, ai-input=yes

# Search/answer crawlers are allowed everywhere
# (the `User-agent: *` group above already covers them — no crawler-specific
# group needed for that side). Training crawlers below are each disallowed
# on the RESULT surfaces (report pages, the server directory, on-demand
# trial results, signed evidence, the badge image, the API, and the MCP
# endpoint), plus the functional pages and static assets listed in the
# "Eight training groups" paragraph below, and allowed everywhere else,
# including every marketing/
# informational page (/, /pricing, /methodology, /about, /maintainers,
# /blog, /probe, /contact, /terms, /privacy, and their /zh/ counterparts) —
# none of those paths appear below, so they fall through to each group's
# implicit allow. This does not affect Search indexing or ranking for any of
# these crawlers' sibling search products (Google-Extended vs Googlebot,
# GPTBot vs OAI-SearchBot, etc. are already separate user-agents by each
# vendor's own design).
#
# IMPORTANT — RFC 9309 has NO GROUP INHERITANCE: a crawler obeys ONLY the
# single most-specific group whose `User-agent` line matches it; it does
# NOT also apply the `*` group's rules on top. So each of the eight named
# groups below repeats the SAME FOUR private-surface Disallow lines
# (/auth/, /account, /zh/auth/, /zh/account) that the `*` group above
# already states — without that repetition, a named-group crawler (e.g.
# GPTBot) would read only its own group's rules, never fall back to `*`,
# and end up ALLOWED on /auth/ and /account, silently un-doing the `*`
# group's disallow for every training crawler. This is NOT redundant
# copy-paste to clean up — do not delete these four lines from any group
# below on the assumption `*` already covers it.
#
# Eight training groups, ONE rule set. Beyond the result surfaces and the
# private surfaces above, every one of the eight groups below also
# disallows (a) the functional pages /signin, /billing, /claim and /keys
# (plus their /zh/ counterparts) and (b) the static assets and icons
# (/assets/, /og, /favicon, /apple-touch-icon). Neither set has any value
# for training, and keeping the eight groups identical is easier to maintain
# (and to test) than a rule set that differs per crawler. /og is a bare
# prefix on purpose: it covers /og-en.png, /og-zh.png and everything under
# /og/. Google-Extended and Applebot-Extended are only opt-out switches
# (the crawlers that actually fetch pages are Googlebot and Applebot), so
# adding these lines to them does not stop those crawlers from rendering
# pages.
#
# The `*` group above deliberately does NOT block the functional pages: those
# pages carry a noindex tag, and a search engine can only see a noindex if it
# is allowed to fetch the page. Blocking them there could leave a bare URL
# with no content in an index instead of nothing at all.
User-agent: GPTBot
Disallow: /check/
Disallow: /zh/check/
Disallow: /servers
Disallow: /zh/servers
Disallow: /try/
Disallow: /zh/try/
Disallow: /attestations/
Disallow: /zh/attestations/
Disallow: /badge/
Disallow: /api/
Disallow: /mcp
Disallow: /auth/
Disallow: /account
Disallow: /zh/auth/
Disallow: /zh/account
Disallow: /signin
Disallow: /zh/signin
Disallow: /billing
Disallow: /zh/billing
Disallow: /claim
Disallow: /zh/claim
Disallow: /keys
Disallow: /zh/keys
Disallow: /assets/
Disallow: /og
Disallow: /favicon
Disallow: /apple-touch-icon

User-agent: ClaudeBot
Disallow: /check/
Disallow: /zh/check/
Disallow: /servers
Disallow: /zh/servers
Disallow: /try/
Disallow: /zh/try/
Disallow: /attestations/
Disallow: /zh/attestations/
Disallow: /badge/
Disallow: /api/
Disallow: /mcp
Disallow: /auth/
Disallow: /account
Disallow: /zh/auth/
Disallow: /zh/account
Disallow: /signin
Disallow: /zh/signin
Disallow: /billing
Disallow: /zh/billing
Disallow: /claim
Disallow: /zh/claim
Disallow: /keys
Disallow: /zh/keys
Disallow: /assets/
Disallow: /og
Disallow: /favicon
Disallow: /apple-touch-icon

User-agent: Google-Extended
Disallow: /check/
Disallow: /zh/check/
Disallow: /servers
Disallow: /zh/servers
Disallow: /try/
Disallow: /zh/try/
Disallow: /attestations/
Disallow: /zh/attestations/
Disallow: /badge/
Disallow: /api/
Disallow: /mcp
Disallow: /auth/
Disallow: /account
Disallow: /zh/auth/
Disallow: /zh/account
Disallow: /signin
Disallow: /zh/signin
Disallow: /billing
Disallow: /zh/billing
Disallow: /claim
Disallow: /zh/claim
Disallow: /keys
Disallow: /zh/keys
Disallow: /assets/
Disallow: /og
Disallow: /favicon
Disallow: /apple-touch-icon

User-agent: Applebot-Extended
Disallow: /check/
Disallow: /zh/check/
Disallow: /servers
Disallow: /zh/servers
Disallow: /try/
Disallow: /zh/try/
Disallow: /attestations/
Disallow: /zh/attestations/
Disallow: /badge/
Disallow: /api/
Disallow: /mcp
Disallow: /auth/
Disallow: /account
Disallow: /zh/auth/
Disallow: /zh/account
Disallow: /signin
Disallow: /zh/signin
Disallow: /billing
Disallow: /zh/billing
Disallow: /claim
Disallow: /zh/claim
Disallow: /keys
Disallow: /zh/keys
Disallow: /assets/
Disallow: /og
Disallow: /favicon
Disallow: /apple-touch-icon

User-agent: meta-externalagent
Disallow: /check/
Disallow: /zh/check/
Disallow: /servers
Disallow: /zh/servers
Disallow: /try/
Disallow: /zh/try/
Disallow: /attestations/
Disallow: /zh/attestations/
Disallow: /badge/
Disallow: /api/
Disallow: /mcp
Disallow: /auth/
Disallow: /account
Disallow: /zh/auth/
Disallow: /zh/account
Disallow: /signin
Disallow: /zh/signin
Disallow: /billing
Disallow: /zh/billing
Disallow: /claim
Disallow: /zh/claim
Disallow: /keys
Disallow: /zh/keys
Disallow: /assets/
Disallow: /og
Disallow: /favicon
Disallow: /apple-touch-icon

User-agent: Amazonbot
Disallow: /check/
Disallow: /zh/check/
Disallow: /servers
Disallow: /zh/servers
Disallow: /try/
Disallow: /zh/try/
Disallow: /attestations/
Disallow: /zh/attestations/
Disallow: /badge/
Disallow: /api/
Disallow: /mcp
Disallow: /auth/
Disallow: /account
Disallow: /zh/auth/
Disallow: /zh/account
Disallow: /signin
Disallow: /zh/signin
Disallow: /billing
Disallow: /zh/billing
Disallow: /claim
Disallow: /zh/claim
Disallow: /keys
Disallow: /zh/keys
Disallow: /assets/
Disallow: /og
Disallow: /favicon
Disallow: /apple-touch-icon

User-agent: CCBot
Disallow: /check/
Disallow: /zh/check/
Disallow: /servers
Disallow: /zh/servers
Disallow: /try/
Disallow: /zh/try/
Disallow: /attestations/
Disallow: /zh/attestations/
Disallow: /badge/
Disallow: /api/
Disallow: /mcp
Disallow: /auth/
Disallow: /account
Disallow: /zh/auth/
Disallow: /zh/account
Disallow: /signin
Disallow: /zh/signin
Disallow: /billing
Disallow: /zh/billing
Disallow: /claim
Disallow: /zh/claim
Disallow: /keys
Disallow: /zh/keys
Disallow: /assets/
Disallow: /og
Disallow: /favicon
Disallow: /apple-touch-icon

User-agent: Bytespider
Disallow: /check/
Disallow: /zh/check/
Disallow: /servers
Disallow: /zh/servers
Disallow: /try/
Disallow: /zh/try/
Disallow: /attestations/
Disallow: /zh/attestations/
Disallow: /badge/
Disallow: /api/
Disallow: /mcp
Disallow: /auth/
Disallow: /account
Disallow: /zh/auth/
Disallow: /zh/account
Disallow: /signin
Disallow: /zh/signin
Disallow: /billing
Disallow: /zh/billing
Disallow: /claim
Disallow: /zh/claim
Disallow: /keys
Disallow: /zh/keys
Disallow: /assets/
Disallow: /og
Disallow: /favicon
Disallow: /apple-touch-icon

Sitemap: https://mcpcheckup.com/sitemap-index.xml