A bug that only appeared in parallel: four servers checked twice, a fraction of a second apart
Until 2026-10-07, a Cloudflare cron call every five minutes took the servers that were due and checked them one after another. That day we made it parallel. The same call now also sends 19 messages to a queue, each message starts an invocation of its own that takes up to five servers, and every invocation checks up to three servers at a time.
The first parallel version went live at 20:53 UTC. Its first round finished in 43 seconds instead of about three minutes. It also checked two servers twice, each a fraction of a second after the first check finished, and the next round did the same to two more. We rolled back seven and a half minutes after the deploy. The bug was older than that day. It only needed a second process to find it.
What claiming means
Every server has a next check time. Once it has passed, the server is due. A round takes due servers by calling one database function, which picks them and creates a job row for each. That row is the claim. It records when the server was taken and, later, when the work finished.
The function skips any server that already has an open job: one with no finish time, created in the last 15 minutes. This in-flight guard keeps two rounds from checking the same server at once. The function also takes a lock, so only one claim runs at any moment.
When a job ends, the probe makes two writes. It marks the job finished, and it moves the server's next check time forward, 12 hours for most servers. They are two separate requests, not one transaction.
The gap between two writes
The order used to be: mark the job finished, then move the next check time. Between those two writes the server had no open job, and its next check time had already passed. To the claim function it looked exactly like a server nobody had checked yet.
With one loop, nothing looked during that gap. The loop making the two writes was the only thing claiming, and it did not claim its next batch until every job in the current one had made both writes. In the seven days before the parallel version, no server had two jobs within ten minutes.
With the fan-out, a round has 20 invocations, the cron call plus 19 from the queue, and each claims whenever it has room. The lock stopped two claims from overlapping. It did nothing about a claim that arrived while another invocation sat between its two writes, a gap about one network round trip wide.
| round (UTC) | jobs | runs | servers | second claim, after the first job finished |
|---|---|---|---|---|
| 20:55 | 24 | 24 | 22 | 0.168 s, 0.236 s |
| 21:00 | 24 | 24 | 22 | 0.118 s, 0.112 s |
All four are servers we list without an active maintainer claim. Our page for operators tells them to expect one round of at most 8 requests every 12 hours. Each got two rounds within seconds.
How we saw it
After a probe deploy we read the first rounds back from the database. Most of those checks compare totals: jobs against runs, runs missing their attestation envelope, jobs left open. All agreed, because each duplicate job had produced its own complete run. The check that caught it counts per server: no server should have two jobs within ten minutes. In the 20:55 round, 24 jobs covered 22 servers.
Why we rolled back instead of patching
The release plan named two ways back before the deploy: the previous version, or the fan-out switched off. Both were approved with the deploy. Going back was one command, run at 21:00:52 UTC.
A fix meant a code change, a test that could fail, a review and another deploy, which took 72 minutes in the end. Any round in that time could have checked more servers twice, and the 21:00 round did, less than 15 seconds before the rollback landed. An hour of slower rounds for us, or duplicate traffic to other people's servers: we chose the slower rounds.
The rollback did not touch the database. The four extra runs are real observations, made seconds apart, and they stay.
The fix
The fix swaps the two writes. The probe now moves the next check time first and marks the job finished second. While the first write is in progress the job is still open, so the guard holds. Once it lands, the server is no longer due. At no point is a server both due and without an open job.
If moving the next check time fails, the probe retries once. If that fails too, it still marks the job finished and stops claiming for the rest of its round, and it logs the error. That error path is the only place the old gap can reopen. If the move matches no row, because the server's status or plan changed mid-round, the job is finished as usual and the server is checked again under its new schedule, as intended.
If the probe now stops between the two writes, the server already has its next check time and the job stays open. Nothing checks it twice, and the open job shows on /status.
Two tests came with the fix. One records the order of the database calls and asserts that the move comes first. The other puts a second claimer between the two writes and asserts that it gets nothing. With the order swapped back, both fail.
The cleaner fix is one database transaction for both writes. That needs a migration and is queued; swapping the order was the smallest change that closed the gap the same day. The fixed version went live at 22:13:24 UTC. Its first two rounds had 21 jobs, 21 runs and 21 servers each.
What we take from it
The old order was correct as long as one loop did all the claiming. The code even guarded against claiming a finished job's server straight back, but only within one invocation, and only when the second write had failed. The change that broke the one-loop condition was two lines of configuration: queue messages per round from 0 to 19, and servers checked at a time from 1 to 3. The gap was in code neither line touched.
When something becomes parallel, look for pairs of writes that only make sense together. In a sequential system nothing runs between them. In a parallel one, something will.
How we checked
The population is every probe job requested from 22:13:24 UTC on 2026-10-07, when the fixed version went live, to 13:15:31 UTC on 2026-10-08: 3,851 jobs for 2,980 servers. A duplicate claim is a job requested less than ten minutes after the previous job for the same server. Our most frequent schedule is every 15 minutes, so such a pair is never a scheduled check; a server whose maintainer claims it mid-round is re-checked straight away by design and would also count, so the definition can only overcount. There were 0. The closest two jobs for one server were 13 minutes 35 seconds apart. The same definition finds 0 in the seven days before the parallel version and 4 while it was live. Table gaps are the second job's request time minus the first job's finish time. Every figure is a read-only query against the production database.