Tagged: engineering
2026-10-17From one loop to nineteen: we measured Cloudflare Queues before running our checks on itOur probe checked servers one at a time. We measured how queue consumers ramp up, fitted a model, tested its predictions, then went to 19 messages a round.2026-10-15The round the platform cancelled and ran again, and 52 minutes of HTTP 503 on /statusA cancelled probe round leaves a job that never finishes, and /status reported it as stuck for 52 minutes. What counts as stuck now, and what we kept.2026-10-11Seven thousand tool snapshots, copied to cold storage byte for byteWe copied 7,102 tool snapshots from our database into gzip objects in cold storage. How we checked the copies, and what has not been copied yet.2026-10-10A bug that only appeared in parallel: four servers checked twice, a fraction of a second apartOur first parallel probe rounds checked four servers twice, 0.11 to 0.24 seconds apart. Why one loop never hit it, why we rolled back, and the fix.2026-10-09The words our blog will not store, and why this post cannot name themSince 2026-10-08 new posts on our blog live in a database whose trigger refuses to publish any post holding a verdict word, this one included.2026-09-30Five hours when our probe ran nothing, and nothing told usOn 2026-09-23 the scheduler that starts our probe every five minutes stopped calling it. An unrelated deploy brought it back five hours later, and we only noticed nearly four hours after that, reading hourly counts. What happened, what we can and cannot say about why, and the two signals we added.2026-09-30A rate that could only ever say 100 percentOn 2026-09-23, all 833 servers in our directory showed the same reachability figure: 100 percent. Not because they had all answered us every day, but because of how we counted. We found it while fact-checking a blog post. What was wrong, what we changed, and what the figure shows now.2026-09-24The null byte that would not storeOne tracked MCP server published a legal regular expression containing \u0000 in a tool schema. PostgreSQL cannot hold that character, so for four days every snapshot of that server failed after its header row was written. How we fixed it without rewriting the evidence.2026-09-24The probe round that was cancelled at 67 secondsThree probe rounds died halfway through, and the servers left behind were not re-checked for half an hour. The cancellation was the platform doing something ordinary. The half hour was our own guard. How the window went from 30 minutes to 15, and why not to 10.2026-09-24Sign-in was broken for two days and nothing logged itFrom 12 to 14 September neither Google sign-in nor the email link worked on mcpcheckup.com. Every server-side log was clean. The cause was a build with two empty variables, made from a clean checkout on purpose. What the fix was, and the gate that now refuses to deploy that build again.2026-09-24Three ways our first chart was wrong before it shippedThe directory statistics page has one small line chart: signed results per day, last 90 days. In review it turned out to cover the last 90 rows, not days; its timestamps could be read in the wrong time zone; and on a wide screen the line did not sit over its labels. All three fixed before merge.2026-09-23A test that failed two percent of the timeAn open-redirect guard in our test suite went red at random, about one run in fifty. The cause was a substring check against a 43-character token. The first fix would have made the guard blind. The lesson is about what a guard's comment owes the reader.2026-09-23The spinner that never stopped: back/forward cache and a waiting state with no way homeSubmit the homepage check, press Back, and the button came back disabled, its label frozen mid-count, with nothing able to re-enable it. The cause was the browser's back/forward cache restoring a page mid-wait. The fix: a small controller with a pageshow reset, and removing a beforeunload listener.2026-09-23Our CI builds the site with none of its keys, on purposeType checks and unit tests were green while a stray test file in the pages directory would have broken the production build. So CI now runs a real astro build, and injects no secrets to do it, because no prerendered route is allowed to read the database at build time.2026-09-14Every production migration now runs twice. The first time, we roll it back.How MCP Checkup applies schema changes to a live Postgres: a full dry run inside one transaction that ends with a deliberate exception, then the real apply. What the dry run measures, and the two things it caught this month.