The round the platform cancelled and ran again, and 52 minutes of HTTP 503 on /status

Our probe runs on Cloudflare Workers. Every five minutes a cron trigger calls it, and each call takes the servers that are due and checks them. Now and then the platform cancels a call partway through and delivers the same cron event again a moment later. An earlier post covered how long the servers left behind had to wait. This one is about what a cancellation did to our public status page.

What a cancelled round leaves behind

When a round takes a server, it writes a job row with the time the job was requested. When the work ends, the probe writes a finish time. A cancelled call never gets that far, and nothing goes back to close the row.

The redelivered call does not take the same server straight away: a server with an open job younger than 15 minutes counts as in flight, so that two calls never check it at once. After those 15 minutes, a later round checks the server and finishes a new job. The old row is still open.

What /status did with it

/status says whether the probe itself is running, and outside monitors poll it. One of its rules: if a job requested in the past hour has stayed unfinished for more than 8 minutes, the page returns HTTP 503. Until 2026-10-08 that rule did not ask whether the server had been checked again since. An open row from a cancelled round kept the page at 503 from the row's 8th minute to its 60th, when it left the one-hour window: 52 minutes.

On 2026-10-07 that happened twice in about two hours.

• 15:50 UTC. The platform cancelled the call after 69.3 seconds of wall time and 205 milliseconds of CPU, with 8 servers done. A call with the same scheduled time started at 15:51:18 and checked 11 others. Two jobs requested at 15:50:54 were left open. /status returned 503 from 15:58:55 until about 16:51. Both servers had been checked again by 16:07:03.
• 17:55 UTC. The platform cancelled the call after 150.5 seconds. One job, requested at 17:57:09, had not even started. The redelivered call began at 17:57:39. /status returned 503 from 18:05:12 until about 18:57. That server had been checked again by 18:12:50.

Each time, for about 44 minutes, the page said a job was stuck when its server had already been checked again.

Why we did not just stop counting them

The simplest change would have been to stop counting jobs like these. We did not, for two reasons. First, the 503 was not false: a job had been requested and had not finished. Second, an open job is also what a real fault looks like. A probe that takes a job and then hangs or crashes before writing a result leaves the same row behind. The page's other signals can miss that case for up to an hour, which is why the stuck check exists. A rule that hid open jobs would hide that case too.

So the question was narrower. An open job whose server a newer job has since checked and finished says the probe is working. An open job with nothing newer behind it says we do not know.

The rule now

Since 2026-10-08, an unfinished job counts as stuck only when all three hold:

• it was requested in the past hour;
• it has stayed unfinished for more than 8 minutes since it was requested;
• no newer job for the same server has finished since.

A job that meets the first two but not the third is counted separately, as abandoned and redone. Only stuck jobs make /status return 503. "Newer" means requested later. A newer job that has been taken but not finished does not count, and neither does a finished job for a different server. The 8 minutes and the one-hour window did not change.

What /status shows now

The page always shows both counts, including when they are 0. The first line reads "Probe jobs unfinished after 8 minutes, with no newer job for the same target finished since (past hour)", followed by the count. The second reads "Probe jobs unfinished after 8 minutes, but a newer job for the same target has since finished (past hour)", followed by its count.

When the first count is above 0, the page returns 503 and says why: "At least one probe job requested in the past hour has stayed unfinished for more than 8 minutes, and no newer job for the same target has finished since. This page returns HTTP 503 while that is the case."

These lines used to say "claimed". The time the page shows is when the job was requested, and the 17:57 job was requested but never started, so they now say "requested".

The window that remains

After a cancellation, the server cannot be taken again until 15 minutes after the open job was requested. So /status still returns 503 from the 8th minute until the newer job finishes, usually between the 15th and the 20th minute. Under the new rule, the 17:57 job gives 7 minutes 40 seconds of 503, from 18:05:09 to 18:12:50, instead of 52 minutes. The two jobs from 15:50 give 8 minutes 9 seconds.

We kept that window. The 15-minute guard sets it, and the alternative, counting a newer job that is still running as enough, would have changed the rule itself to save a minute or two. An uptime monitor may still alert once after a cancellation. That alert describes something that did happen.

The case the rule does not cover

Since another fix on 2026-10-07, the probe moves a server's next check time forward before it marks the job finished. If it stops between those two writes, or the second write fails, the run is written and the next check is 12 hours away, but the job stays open. No newer job comes within the hour, so it counts as stuck until it leaves the window.

We have seen one such job. A job requested at 05:11:09 UTC on 2026-10-08 has a run that started at 05:11:09.632 and a server whose next check had moved to 17:10, but no finish time. The Worker's log for that call shows the second write failing: at 05:11:30.803 the probe logged that the update setting the finish time had come back with error code 522, a connection timeout, and it went on with the round. The call ended normally; the platform did not cancel it. The new rule, which went live later that day, counts the job the same way.

Since 2026-10-01, seven rounds have left 50 jobs open. In six, every server was checked again by a newer job that finished within 24 minutes of the open job's request. The seventh is the 05:11 job.

How we checked

/status reads one database function that returns counts and times only, nothing about any single server; the rule above is its definition since 2026-10-08. The times for 2026-10-07 come from the job table, the Worker's log stream for those two rounds and our own polling of /status that afternoon. The 7 minutes 40 seconds is the new function run against production data in a transaction that was rolled back. The figures since 1 October cover every job requested from 2026-10-01 00:00 UTC that had no finish time and was over an hour old at 13:17 UTC on 2026-10-08; a job is superseded when a job for the same server, requested later, has a finish time. The causes of the seven rounds were not all traced; the two on 2026-10-07 are the ones where the platform's records show a cancelled call, and the cause for the 05:11 job comes from the Worker's log for that call. Every database figure is a read-only query.