Five hours when our probe ran nothing, and nothing told us

Our probe is a Cloudflare Worker with a cron trigger. Every five minutes the platform calls it, and each call claims the servers that are due, up to 30, probes them and writes the results. All claiming and probing of due servers starts from that call; nothing else starts it. An on-demand check can probe a single server when someone asks for one, but its result is never signed, never becomes a run, and never enters the directory. If the platform stops calling, the directory stops being observed.

What happened

At 11:42 UTC on 2026-09-23 we deployed a new version of the probe. The platform called it once, at 11:45:47. That call claimed 28 servers and finished at 11:47:48.

Then nothing. Between 11:50 and 16:45 the schedule should have produced 60 calls. The platform’s own invocation log for the Worker shows none.

At 16:47:16 we deployed again. It was a planned release of check suite 0.7.0 and had nothing to do with the gap. The first call came three minutes later, at 16:50:58. The probe then worked through the backlog, 90 runs per quarter hour instead of 30, until 18:45, and returned to its usual pace at 19:00.

We noticed none of this while it was happening. At 20:43 UTC, nearly four hours after it had ended, an hourly count of runs showed no rows at all from 12:00 to 15:59. The same gap is why, in an article we published that week, 143 of the 1,299 servers then in the directory had no run in the window we used.

What we can say about why

• Our code did not fail. There was no call for it to fail in, and the last call before the gap ran to completion.
• The database was not stuck. No job from that day was left in flight, and the database’s own scheduled jobs ran on time throughout.
• Cloudflare’s status page listed no incident for Workers or Cron Triggers on 22 or 23 September.
• The gap began right after one deploy and ended right after the next.

Our best reading is that the cron trigger for the 11:42 version was not delivered on the platform side, and deploying again registered it afresh. That is an inference. From our side we cannot prove more, and we have not written code to work around a cause we cannot confirm.

What we added

Two signals. The status page has been live since 2026-09-23; the heartbeat went live the next day, 2026-09-24 at 17:08 UTC.

A heartbeat to an outside service. Each five-minute call of the probe sends one ping when it starts. It sends a second ping only when a probing cycle has run to its end: not when the cycle was refused or threw an error. A dead-man’s switch at Healthchecks.io emails us when the pings stop. A scheduler that goes quiet now means missing pings, and missing pings mean an email, instead of a number someone happens to read hours later. The address the probe pings is a secret held by the Worker. It is not in our repository.

A public status page. /status shows the latest probe run, the number of runs in the last 60 minutes, and how many due servers have not yet been picked up. It reads one database function that returns a single aggregate row, with no data about any individual server. When there has been no run for 20 minutes, it says so in one line, unless nothing is due and a run happened within the last hour.

Behind both is a short written procedure. If either signal fires and the platform shows no scheduled call in the last 15 minutes, we redeploy the current version. If calls still do not appear within five minutes, we open a ticket with Cloudflare and attach the invocation log and our run counts.

The heartbeat does not tell us that the probing produced sound results. It tells us the Worker was called and the cycle ran. Whether results are fresh is what /status reads from the runs themselves.

Since then

From 2026-09-24 at 17:08 UTC, when the heartbeat went live, through 2026-09-30, the longest gap between two consecutive runs starting was 14 minutes and 4 seconds. No gap exceeded 20 minutes.

We publish results that say when they were observed, so a gap in observing is itself something a reader should be able to see. This one was only visible in hindsight. The two signals are there so that the next one is visible while it is happening.

Where the numbers come from

The call times are from the Worker’s invocation log in the Cloudflare dashboard, read on 2026-09-23 for the window 11:40 to 16:55 UTC. The run and job counts are read-only queries against the production database, grouped by hour and by quarter hour. The status-page check is Cloudflare’s public incident feed, read the same evening. The figures since 2026-09-24 are one read-only query over the start times of all non-trial runs since the heartbeat deploy.