The null byte that would not store

On 2026-09-21 at 03:48 UTC the probe wrote a tool snapshot header for one tracked server, got 201 Created, then sent the snapshot’s items and got 400. The database log for the same second says why:

ERROR 22P05 unsupported Unicode escape sequence
DETAIL  \u0000 cannot be converted to text.
CONTEXT JSON data, line 1: …{"allOf":[{"type":"string","pattern":"^[^\u0000…

One of the server’s tools declares an input schema whose pattern is a regular expression beginning ^[^\u0000. That is a legal regex and a legal JSON string. It is also a character that PostgreSQL will not store in a text or jsonb column, and PostgREST refuses the request while it is still parsing the body, before any SQL runs.

What it did to the page

The header row had already committed, so every round left behind a snapshot with zero items. With no complete snapshot, that server never got a latest-snapshot pointer, never got a signed envelope for those runs, and never appeared in the directory. It was probed twice a day from 2026-09-17 to 2026-09-21 and nine orphan header rows accumulated, one per round.

Two earlier guesses about this server, duplicate tool names and CPU time, were both wrong. The log line above is what settled it, which is a good argument for reading the log before theorising.

Why this is not the server’s problem

Nothing about the tool is malformed. \u0000 inside a character class is an ordinary way to say “anything but NUL”. The limit is ours: our storage cannot hold that character, and the probe was sending it anyway.

Measuring what the database actually rejects, with pg_input_is_valid against production, gave a precise set:

input	jsonb accepts it
"\u0000"	no
"\ud800" (lone high surrogate)	no
"\udc00" (lone low surrogate)	no
"😀" (a surrogate pair)	yes
"\u0001"	yes
"￿"	yes

So “unstorable” means exactly U+0000 or an unpaired UTF-16 surrogate, and nothing else. JavaScript strings may legally contain a lone surrogate, and JSON.stringify writes it out verbatim, which is how one would reach the wire.

Two fixes we refused

The obvious fix is to strip the character before writing. We did not do that. The report page shows hashes and fingerprints of what the server published, and a reader is entitled to recompute them from the server’s own tools/list. If we quietly rewrote the text, the stored text and the hash would no longer describe the same thing.

The second fix we sketched, then withdrew, was writing a marker object into the schema column itself, something like {"$unstorable": …}. The schema column is third-party text. A server could publish that marker on its own, and any reader keying off it would be repeating the server’s claim as if it were ours.

What we did instead

Each item row now carries a column, unstorable_fields, that lists which of its three text columns (tool_name, description_raw, input_schema_raw) could not be stored. The column that could not be stored is set to NULL. Two database constraints keep the declaration honest: the array may only name those three columns, and a column it names must actually be NULL. The public read roles have column-level SELECT on this table and no write privileges at all, so the declaration can only come from our probe.

The order of operations in the probe is the load-bearing part. Every hash on the page, name_hash, schema_hash, the toolset fingerprint and the snapshot’s content digest, is computed over the rows as observed. Only after that does the probe make a separate copy for the insert with the unstorable columns nulled. A comment in the code says why the order is load-bearing.

When it is the tool’s own name that cannot be stored, the row gets a placeholder name built from the name’s hash, unstorable: followed by sixteen hex digits, and unstorable_fields says so. A server cannot fake that state by naming a tool the same way: its unstorable_fields would be empty.

The report page reads the declaration first. Where a description could not be stored it prints:

Not saved: the observed text contains a character this database cannot store (U+0000 or an unpaired surrogate).

Where a schema could not be stored, the schema line reads (input schema not saved: the observed schema contains a character this database cannot store). Where a name could not be stored, the placeholder is explained, and the fingerprint section adds one more sentence so nobody assumes the hashes were computed over the placeholder:

Hashes and fingerprints on this page are computed over the text as observed, not over this placeholder.

That last sentence appears only when a placeholder name is actually on the page.

Two smaller things in the same change

The orphan rows should never have existed. Items are now written in one insert, and if that insert fails for any reason the probe deletes the header row it just wrote and rethrows the original error. Both foreign keys onto the header cascade, so there is no half-written state to clean up.

The detection itself walks the schema as a structure, keys included, because in jsonb a key is text too. It uses an explicit stack rather than recursion and has no depth limit: a clean schema two hundred levels deep must go in unchanged. The leaf test is written out over UTF-16 code units rather than as a regex. The regex /[\u0000\ud800-\udfff]/ looks right and is wrong: it matches both halves of a well-formed pair, so every emoji in a description would be declared unstorable. The correct spelling is /\u0000|\p{Cs}/u, and the tests pin both facts.

The result

The migration adding the column was dry-run inside a transaction against production and rolled back, then applied at 13:58 UTC on 2026-09-21. The probe and the site were deployed after it. At the server’s next scheduled round, 15:48 UTC, the snapshot went in whole: 11 tools, two of them with input_schema_raw declared unstorable and set to NULL, the descriptions kept, a signed envelope, and a directory row. The nine orphan header rows were deleted afterwards, again dry-run first. The database log has shown no 22P05 since.

Across the whole database, two item rows carry a non-empty unstorable_fields. Out of about 1,300 tracked servers, exactly one needed this. That is roughly the ratio we expected, and it is still worth the column: the alternative was a server that could never be listed, for a reason nobody could see.

Where the numbers come from

Read-only queries against the production database on 2026-09-21, before and after the fix: the orphan header count (tool_snapshots rows with tool_count > 0 and no items, nine before, zero after), the recovered snapshot (11 items, two with a non-empty unstorable_fields), the 22P05 count in the PostgreSQL log since the probe deployment (zero), and the pg_input_is_valid table above. The migration’s own dry run validated the two constraints against all 46,395 existing item rows, none of which needed the new column.