sidekiq concurrency with many threads

Question

We use Sidekiq to process background jobs, such as this:

def process_order(id)
... do something
end

Sidekiq is set to 50-100 threads, so we've seen process_order run on the same id at the same time, which we don't want. What is the best way to ensure process_order do not run on the same id concurrently?

I looked into https://ruby-concurrency.github.io/concurrent-ruby/master/Concurrent/Async.html but the example isn't very clear (for a newbie).

Mike Perham · Accepted Answer · 2024-09-12 19:03:37Z

0

You're looking for the unique jobs feature. Sidekiq Enterprise has it and there are 3rd party plugins which also provide it.

50-100 threads is way too much. Ruby doesn't scale well to that many threads.

answered Sep 12, 2024 at 19:03

Mike Perham

22.4k6 gold badges64 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

sidekiq concurrency with many threads

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related