Seven thousand tool snapshots, copied to cold storage byte for byte

A tool snapshot is a server's tool list as our probe stored it at one point in time: every tool's name, description and input schema, plus the fingerprints computed from them. On 2026-10-08 our database held 7,111 snapshots with 116,621 tool entries between them. The table that holds those entries was 271 MB, the second largest in an 853 MB database.

Between 18:37Z on 2026-10-07 and 12:37Z on 2026-10-08 we copied 7,102 of those snapshots out of Postgres into object storage, one gzip object per snapshot. This post is about how we know the copies are exact, and about what has not been copied.

Why bother

We monitor 2,980 endpoints today, and the directory keeps growing. Pages read each server's current tool list. Older ones are history: they matter, and they are rarely read. Left in the database, that history grows with every server we add.

Copying is not moving. The copying deleted nothing: all 7,111 snapshot rows are still in the database. Removing older rows is a separate, later step that has not run, and it depends on copies we can prove.

What one copy looks like

Each snapshot becomes one object named after its id: cold/v1/snap/<snapshot id>.json. Inside is one JSON document with five fields: a schema name (cold/v1), the encoding (gzip), the run that produced the snapshot (11 columns), the snapshot row itself (9 columns), and every tool entry (10 columns each, ordered by id). The bytes are gzipped before upload. The key is built only from a lowercase UUID; anything else is refused before any request goes out.

Before uploading, the writer hashes the JSON bytes with SHA-256. After uploading, it asks storage for the object's size and compares it with what it sent. Only then does it write three numbers into the snapshot's row: cold_bytes (the size as stored), cold_raw_bytes (the size before gzip) and cold_sha256 (the hash). That update only applies while the row has no copy recorded, so a second writer can never overwrite the first.

Across the 7,102 copies, 192,740,260 bytes of JSON became 48,749,733 bytes in storage, a ratio of 3.95 to 1. The median snapshot is 8.6 KB of JSON and 2.6 KB stored.

Four ways we checked

Numbers that would not survive the trip. JavaScript reads every JSON number as a double. Text like 1.50 comes back as 1.5, a number with 18 significant digits loses some of them, and -0 is written back as 0. Before the first copy was written, we looked at every number inside every stored input schema: 233,554 of them. None had a trailing zero after the decimal point, none had 18 or more significant digits, and none was negative zero. 24,357 had 16 or 17 significant digits, but they were only 10 distinct values, all already in the shortest form JavaScript writes. We also decided to compare copies by value and never by text, because a number of 1e21 or more comes back in exponent form: the same value in different characters.

A hash at write time. The SHA-256 of the exact JSON bytes sits in the row. Anyone who later reads an object can unzip it, hash it, and know whether it is the object we wrote.

A proof line every hour. The copy job runs once an hour, at minute 37. Before it copies anything, it picks one archived snapshot at random and reads its object through the same code that serves cold reads. That code checks three things: the first two stored bytes are 1f 8b, the gzip signature; the object unzips without going past a 1 MiB cap; and the SHA-256 of the result equals cold_sha256. The job does the same for one archived run object, the kind described further down, then prints a single line. From 19:37Z on 2026-10-07, the first round with snapshot objects to sample, through 12:37Z the next day, all 18 rounds printed ok for both.

Three copies read back by hand. We downloaded three objects with the storage command-line tool and hashed them locally. All three hashes equal cold_sha256, and all three byte counts equal cold_raw_bytes. Then we compared what was inside with the database rows, by value: the run, the snapshot and every tool entry. All three matched.

How fast it went

Within the hourly round, copying comes last. It takes the oldest snapshots first, skips any younger than 15 minutes, and copies at most 500 per round. Like the rest of the round, it starts no new work once 10 minutes have passed.

The first four rounds, from 18:37Z on 2026-10-07, copied 312, 319, 314 and 312 snapshots. Each ran into the 10-minute limit, with a median upload of about 0.8 seconds: about 310 an hour, not the 500 we had planned for. After a probe update later that evening, the median upload took between 0.16 and 0.36 seconds, and each round reached its cap of 500 in under seven minutes. By the 09:37Z round on 2026-10-08 the backlog was down to the three snapshots described below, and later rounds only had new ones to copy. In all, 19 rounds wrote the 7,102 copies.

One upload failed, at 03:40Z, with an error from the storage service. The round counted it and logged it, the row kept no key, and a later round copied that snapshot.

What has not been copied

One snapshot is too large. Its JSON is 1,841,224 bytes, more than the 1 MiB our own reader accepts. We never write an object our reader would refuse, so this one stays in the database, and the round logs it every hour.

Two snapshots list exactly 1,000 tools. The copy job reads a snapshot back from the database before writing it, and that read returns at most 1,000 rows at a time. With exactly 1,000 tool entries, the job cannot tell a complete list from a cut one, so it refuses to write rather than store a copy that might be short. Both stay in the database. Reading in pages is on our list.

New snapshots wait. A snapshot is copied in the first round after it turns 15 minutes old. At 13:15Z on 2026-10-08, six were waiting.

Older signed envelopes. A similar object holds each run: the run, its check results and its envelope, gzipped together under cold/v1/att/<envelope id>.json. Signed envelopes from 2026-09-25 onward get theirs when the run finishes, with one exception on 2026-09-29 whose write failed and was recorded as a warning on that run. The 13,276 signed envelopes from before 2026-09-25 do not have a cold copy yet. Our rule for the later removal step is that a run whose envelope has no checked copy is never a candidate.

Why byte for byte, and not close enough

A signed attestation records two fingerprints of the tool list: one over the tool names and one over each tool's name and input schema. The snapshot is the list those fingerprints came from. If you want to check that a signed fingerprint matches the tools we say we saw, you need those tools exactly. A copy that rounded one number inside an input schema would still look right to a person, and it would no longer reproduce the signed fingerprint. A copy that changed one character of a description would no longer match the snapshot's own content digest, which covers descriptions. Either way, our archive would contradict our own signature, and a reader could not tell which one to believe. What the signature covers has its own post.

How we counted

• Snapshots are every row in our tool snapshot table at 2026-10-08T13:15Z: 7,111. A snapshot counts as copied when its row has a cold object key, which the writer sets only after the upload's size was confirmed: 7,102 did, 9 did not (the three above and six new ones).
• Sizes are sums of the sizes recorded in those rows. Database and table sizes are what Postgres reported at the same time, indexes included.
• Endpoints monitored are those with at least one scheduled run in the seven days before 2026-10-08T13:00Z.
• The 233,554 numbers were counted on 2026-10-07 at 15:37Z, over every input schema stored then.
• Round counts, upload times and the proof lines come from the hourly job's own log lines, 2026-10-07 18:37Z through 2026-10-08 12:37Z.
• The three objects were downloaded and compared on 2026-10-07. We hashed the same files again on 2026-10-08 at 13:15Z.
• Signed envelopes carry a signature and are not disqualified.
• Limits: the hourly proof reads one object of each kind per round, and the comparison by hand covered three objects. We have not re-read all 7,102 objects. What covers every one of them is the write path: hash before gzip, size check after upload, and a row update only after both.