Here’s the SolidQueue batches 2.5 year architectural journey, filled with O(n^2) algorithms, hot row contention, excessive jobs, and much in between.
There’s a strong chance you’ll learn a thing or two - I certainly did 😮💨
Definitely an important question, and something I spent a LOT of time optimizing. I went through multiple iterations of performance and load testing, before landing on this version. I'm planning to do a small blog post on the internals of batches to explain it a bit. /1
Looks really cool. But how large can it scale?
In chronoforge we had to implement adaptive polling to ensure it scales up. But that's at the cost of completion latency.
github.com/radioactive-la…
The docs on SolidQueue batches are… solid 😉
In addition, I wrote up this blog post to dig in a bit further on how to use batches in SolidQueue, including some code inspiration - like splitting a RubyLLM chat into parallel sub agents using batches
When your #rubyfriend@jeremysmithco unintentionally puts you on blast at RailsConf 2025 for not delivering on batches and it spurs you to finally finish it