The probe round that was cancelled at 67 seconds
The probe runs on a Cloudflare Worker with a cron trigger every five minutes. When the rounds below ran, each invocation claimed up to 30 servers that were due, checked them one after another, and logged round complete. Three rounds in September never logged it: one stopped after 7 of 9 servers, one after 15 of 30, one after 27 of 30.
None of them was a timeout. The one round we have the platform’s own record of stopped after 67 seconds of wall time and 564 milliseconds of CPU, nowhere near either limit. The rows in our job table said the servers had been claimed and never finished, and each of those servers was checked again on a later tick, so nothing was lost. But “later” turned out to mean about 30 minutes later, and for a server on the 15-minute tier that is two missed checks.
What actually happened
The answer was not in our logs, because our logs stop where the invocation stopped. It was in the platform’s invocation view. For the round of 2026-09-18 at 20:30:11 UTC there were two scheduled invocations with the same scheduledTime:
| ended at (UTC) | outcome | wall time | CPU time | what it logged |
|---|---|---|---|---|
| 20:31:53 | canceled | 67,443 ms | 564 ms | 15 -> run lines, no round complete, no error |
| 20:32:04 | ok | 117 ms | 3 ms | nothing: it claimed zero servers |
Cloudflare cancelled the first invocation 67 seconds in and, eleven seconds later, redelivered the same cron event. That is ordinary platform behaviour, not a fault. The redelivery did what it should: it asked the database for due servers. It got none.
Why it got none
The claim is a database function with an in-flight guard: a server whose last job row has no finished_at and was requested less than 30 minutes ago is treated as still being probed, so that two overlapping invocations cannot probe the same server twice. The 15 servers the cancelled invocation had claimed and not finished sat exactly inside that description. The guard was doing its job against a case that no longer existed. Those 15 servers were picked up by the first tick after the 30-minute mark, which in practice meant 30 to 35 minutes late depending on where within a tick the claim happened to land.
We saw it again on 2026-09-21 at 16:45:16 UTC, the first round after a probe deployment. Twelve servers were left behind. They were reclaimed by the 17:15 tick, whose claim ran at 17:15:17.043 UTC, which is 88 milliseconds past the guard boundary at 17:15:16.954. Had the claim landed a tenth of a second earlier, the same twelve servers would have waited for the 17:20 tick instead. So under the old guard the penalty was not a number; it was 30 or 35 minutes, decided by jitter.
A note on evidence: the dashboard filter outcome != ok over a seven-day range returned “No events found” for these rounds. The invocation list is sampled, and the page says so. A narrow time window around the tick found them at once.
Shrinking the guard, first attempt
The fix is obvious: make the guard window shorter than 30 minutes, so a cancelled round’s leftovers are reclaimed by the next tick or the one after. The question is how short.
The first draft said 10 minutes, and it came with measurements. Over ten days and 16,332 job rows, the longest a single job had taken from claim to finish was 275.8 seconds; the 99.9th percentile was 249.1 seconds; none had exceeded 5 minutes. Ten minutes was more than twice the worst case ever observed.
A review comment on the pull request showed why that argument was not good enough. The per-request timeout is re-armed for every HTTP request a job makes, and the round-level budget is only checked between requests, never handed down into a request in progress. A single job could therefore spend about 20 seconds in its probing phase alone, before snapshot writes and signing. Thirty jobs in sequence is about 11 minutes. The measurements described the servers we had met; the code described the servers we could meet. A guard set from measurements is a guard that a slower server can walk through.
Fifteen, and why that number
Fifteen minutes is the maximum wall-clock time the platform allows a scheduled invocation. An invocation that is still running at 15 minutes is stopped, so a job row that was requested more than 15 minutes ago and has no finished_at cannot belong to a live invocation. The guard can rely on that regardless of how slow a server is or how the code’s budgets are arranged. It is the smallest whole number of ticks that can be argued from a limit rather than from a sample, so that is the value in the migration, which changes exactly one literal in the claim function, '30 minutes' to '15 minutes'.
The argument has three named premises: the platform enforces the 15-minute limit; a claim happens after its invocation starts; and the database clock and the runtime clock do not drift apart by more than that gap within 15 minutes. The two clocks never need to agree on the time, only on how fast it passes, because each inequality uses differences on one clock.
The rule we wrote down afterwards: a safety window is set from a structural bound or a platform hard limit, never from a measured maximum. The measurements are still useful. They tell us the worst case is far below the bound, which is different from telling us where the bound is.
After the change
The migration was dry-run in a transaction against production and rolled back, then applied on 2026-09-21. The read-only check 24 hours after the probe deployment found one round, at 12:45:53 UTC on 2026-09-22, with 4 of 28 jobs unfinished. All four servers had a new run between 13:01:06 and 13:01:19 UTC, 15.3 minutes later, on the first tick after the guard lapsed. Before the change the same case took at least 30.
There is a further step we have not taken: passing the remaining round budget down into each request, so that a job’s probing phase is bounded by the round and not only by its own per-request timers. That changes how a check can time out, which changes what the check can conclude, so it gets its own round of review. Until then the guard stays at 15. The cancellations themselves we do not intend to fix. They are the platform’s, they are rare, about two in 270 rounds, and a round that can be resumed on the next tick is the right shape for something that can be cancelled at any second.
Where the numbers come from
The two invocation rows are from the Cloudflare dashboard’s invocation view, filtered to the minute around 2026-09-18 20:30 UTC. The job durations (16,332 rows, max 275.8 s, p99.9 249.1 s) and the reclaim timings (17:15:17.042628 UTC against a boundary of 17:15:16.954154 UTC; four servers re-run at 13:01:06 to 13:01:19 UTC on 2026-09-22) are read-only queries against the probe_jobs and runs tables. The per-request and round budget behaviour is read from the probe’s source, not measured.