From one loop to nineteen: we measured Cloudflare Queues before running our checks on it
Until 2026-10-07, one Cloudflare cron call every five minutes did all of our checking. It took the servers that were due and checked them one after another, 21 on a typical round. From its first job to its last finished check, the median round took 116 seconds, and one round in twenty took more than 200. The next round started five minutes after this one began, and the work was already using most of that time.
Since 22:13 UTC on 2026-10-07, each round is spread over 20 invocations. The cron call sends 19 messages to a queue, each message starts an invocation of its own that takes up to five servers, and every invocation checks up to three at a time. Over the first 72 hours, the median round took 23 seconds from first job to last check, and the slowest took 42.
This post is about the step between those two paragraphs. We did not know how fast Cloudflare Queues would start 19 invocations, so we measured that first.
The question the documentation left open
A queue only helps if its messages are processed at the same time. Cloudflare's documentation says Queues decides whether to add consumers only after a consumer finishes a batch. It does not say how quickly. If 19 messages were handled one after another, the parallel version would be slower than the loop it replaced.
So the first version of our queue did no checking. Its consumer accepted one kind of message: a test that called our signing service's public-key endpoint a given number of times and then waited. It claimed no servers and signed nothing. We deployed it at 11:50 UTC on 2026-10-07 and sent it two tests.
The first was one message asking for 40 calls. All 40 succeeded. Every check calls the signing service, and the design needed one invocation to make at least 30 calls. We learned that the limit is at least 40, not what it is.
The second was nine messages at once, each asking the consumer to hold for 60 seconds. The first started about 2 seconds after we sent it. The second started at 63 seconds, right after the first finished. Concurrency grew from there and peaked at five, and the ninth message finished 257 seconds after the push. Nine minutes of holding had taken more than four.
A model fitted to one curve
From that curve we fitted a simple model. Start with one consumer. When one finishes, start the next message at once. About 12 seconds after each finish, allow one more consumer at the same time. Replayed against the test, the model started the nine messages at 2, 62, 74, 123, 135, 135, 147, 183 and 195 seconds, against a measured 2, 63, 77, 123, 137, 137, 147, 183 and 197. It ended at 255 seconds. The test ended at 257.
The model says something useful: concurrency grows with each finished invocation, so short invocations grow it faster. Our original design had nine messages of up to 30 servers each. In the model, each of those invocations ran for about 90 seconds, few finished early, the ramp stayed slow, and at higher load some messages waited in the queue past the point where we drop them as stale. The design we shipped has the opposite shape: 19 messages of at most five servers each, three checked at a time. In the 72 hours below, such an invocation took a median of under eight seconds.
Testing the model before trusting it
A model fitted to one curve may describe that curve and nothing else. Before writing code for the new shape, we wrote down what the model predicted for a second test and then ran it: 40 messages, each making five calls and holding for 20 seconds, roughly the work of one five-server invocation.
| predicted | measured | |
|---|---|---|
| last message started | about 127 s | 126.1 s |
| last message finished | about 148 s | 146.1 s |
| most invocations at once | about 19 | 17 |
All 40 succeeded. The shape held, so the code was written to it, with the number of messages, the servers per message and the checks at a time as configuration values rather than constants.
What did not change for the servers
Parallel rounds mean more of our checks at the same moment, not more checks per server. Each server is still checked on its own schedule, every 12 hours for servers without an active maintainer claim, as our page for operators says. Two rules in the database keep it that way, however many invocations are claiming: a server with an open job is never claimed again, and no host has more than one job in flight. In the 72 hours below, no host ever had two.
Switching it on
The parallel code went live switched off, with the number of messages set to 0, so rounds ran exactly as before. Switching it on was a separate deploy that changed two values: messages per round from 0 to 19, and servers checked at a time from 1 to 3.
The first switch-on, at 20:53 UTC on 2026-10-07, checked four servers twice within a fraction of a second, through a gap between two writes that one loop could never hit. We rolled back after seven and a half minutes and fixed it. That story has its own post. The fixed version went live at 22:13 UTC.
72 hours later
We set the bar before switching on: every job accounted for, no server checked twice, no host with two jobs at once, and the share of signed runs no lower than before. From 22:13 UTC on 2026-10-07 to 22:13 UTC on 2026-10-10:
- 18,446 jobs and 18,446 runs, every run with its attestation record.
- 0 servers with two jobs less than ten minutes apart.
- 0 hosts with two jobs in flight at once.
- Every listed server checked on each of the three days: 2,980 of 2,980.
- Runs per day: 6,149, 6,149 and 6,148, against 6,151 in the 24 hours before.
- Signed runs: 61.8%, 61.8% and 61.7%, against 61.7% in the same 24 hours. The share depends on how servers answer, not on how we schedule them, and it did not move.
- Rounds: from first job to last finished check, a median of 23 seconds and a slowest of 42, against 116 and 214 for the single loop. Measured from the five-minute mark, the median round ended 80 seconds in, against 175.
Two things were not perfect. Two jobs out of 18,446 never recorded their finish time, though both runs were complete and the next job for each server ran normally. One of them belonged to an invocation that Cloudflare cancelled: in those 72 hours the platform cancelled 1 of about 17,500 invocations, against 13 of about 1,000 in the 72 hours before. We cannot see from outside whether shorter invocations are the reason, so we only report the counts. Why the other job's finish went missing, we have not found yet.
What we take from it
The useful measurement was not the one we planned. We sent nine long messages to find out whether invocations ran in parallel at all, and the answer came with a curve. The curve gave us a model, the model told us to make invocations short, and a second test that we wrote down before running told us whether to believe it.
How we checked
A round is all probe jobs requested in one five-minute bin. The single-loop figures cover the 288 rounds from 12:00 UTC on 2026-10-06 to 12:00 UTC on 2026-10-07; the parallel figures cover the 860 rounds from 22:13 UTC on 2026-10-07 to 22:13 UTC on 2026-10-10. Both have a median of 21 jobs per round. Round time is the last job's finish time minus the first job's request time; from the mark, it is the last finish minus the start of the bin. The two measures differ by when a round's first job starts: a median of 34 seconds after the mark for the single loop and 56 seconds for the parallel version. We have not yet worked out why it starts later. A server counts as checked twice if two of its jobs were requested less than ten minutes apart. A host counts as having two jobs in flight if two of its jobs overlap between claim and finish. Signed runs are runs whose attestation is signed rather than recorded as unsigned. The queue tests are the consumer's own log lines; the cancellation counts come from Cloudflare's invocation analytics for our probe worker. Every database figure is a read-only query against production.