July 27, 2026
Queue the chaos
Tokio Gives Progress, Not Ordering: Scheduling 1M Tasks
The app launched a million tiny jobs, and the internet yelled: “Of course it got chaotic”
TLDR: A developer fired off about a million tiny jobs and discovered the system cared more about speed than keeping things in order, so older work could get delayed in weird ways. Commenters were split between sympathetic veterans saying this lesson hurts everyone once and blunt critics saying: Tokio never promised a neat line.
A Rust developer thought they were being clever: take a flood of incoming events, split each one into tons of tiny jobs, and let Tokio—the popular engine running the async code—blast through them at top speed. And to be fair, it did finish the burst on time. But the logs turned into pure reality-TV chaos: newer events were jumping ahead, older work was showing up suspiciously late, and tasks from early events seemed to vanish into the crowd before finally getting attention. The big lesson? Fast does not mean first-come, first-served.
That’s where the comments got deliciously blunt. One camp basically shrugged and said, welcome to the pain. Veteran commenter jandrewrogers delivered the grizzled-engineer version of “kid, we’ve all been there,” arguing that building a scheduler that is both fast and fair is nightmarishly hard in real systems. Another crowd was even less sympathetic: Tokio was never promising orderly lines, they said, and expecting it to behave like a neat checkout queue is the real mistake. The hottest take was essentially: if you want order, pay for it yourself.
The vibe in the thread was half support group, half roast session. The unspoken meme was: you wanted speed, but also fairness, but also simplicity? Pick two. Even the suggested fix, like futures_orchestra, came with a side of community side-eye—sure, you can get the behavior you want, but now you’re paying the complexity tax. In other words: the code worked, the logs looked cursed, and the commenters had a field day.
Key Points
- •The article describes an event-driven Rust service that spawns a Tokio task per event and then additional tasks per user token within each event.
- •Each event can include up to 1,000 user tokens, and a burst of 1,000 events resulted in roughly 1 million spawned tasks.
- •The system completed the burst within the expected time, but logs showed that some tasks from earlier events were first polled much later than their submission order suggested.
- •The author expected out-of-order completion but was surprised by the extent to which earlier tasks drifted before receiving their first poll.
- •The article explains that Tokio’s multi-threaded scheduler uses worker-local queues, a shared global queue, overflow behavior, and work stealing, which causes older and newer runnable tasks to compete without event-level ordering.