Questions tagged [queue]
The queue tag has no summary.
69 questions
0
votes
1
answer
122
views
Matching supersets
In my application I have requests and items, each of which are associated with property tuples (key value pairs, where keys can be repeated) stored in a Postgres database.
The goal is to find a single ...
- 159
-1
votes
1
answer
374
views
Airflow DAG vs custom worker to process queue of videos
I need to process a queue of videos in a scalable way. Processing includes number of tasks that can depend on each other. Some tasks are computationally expensive some not (e.g. transcribe audio, ...
- 1
0
votes
2
answers
4k
views
How to terminate python queue and instruct all consumer-threads to finish their tasks?
I have a multi-threaded application.
There is 1 thread that produces a resource and puts it into a queue and many (not constant amount) consumer-threads that get the resources from the queue.
When ...
- 111
0
votes
1
answer
265
views
Are self-written queues and linked lists really worth it (better than built-in arrays)?
Let's say I need a queue for my project and the language I use does not have a built-in queue structure. So I decided to implement one myself.
So I create an object with elements positions as the ...
- 39
3
votes
4
answers
2k
views
Iterating through stacks and queues?
Are stacks and queues allowed to be iterated over, conceptually?
It “feels wrong” for some reason but I can't come up with a reason why it would be bad to see through the whole thing that's stacked/...
- 159
1
vote
1
answer
584
views
What kind of queue should I use for processing large volume of data?
I have a segment of customers which may range between 1 and 5000. For each customers, I need to do some processing by making some query to the database using the data from the customer. The query is ...
- 121
2
votes
2
answers
1k
views
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
- 83
0
votes
1
answer
1k
views
Scheduling asynchronous jobs with efficient round-robin algorithm using Redis
We are trying to redesign our microservice architecture-based application to support multiple tenants. We have a simple queue service that will be utilized by other services to queue any asynchronous ...
- 101
-1
votes
1
answer
44
views
Options to maintain queue of resources that will be uniquely returned
I need to maintain a queue of resources in a cache such that when multiple clients try to pop elements from the queue concurrently, each element would be uniquely returned to only one of the clients. ...
- 157
-1
votes
1
answer
153
views
Design approach to decouple DB write to reduce application latency
I want to use DynamoDB to save application operation data at the end of a session for auditing purposes without affecting the actual application latency. Because this date is only for auditing so we ...
0
votes
0
answers
38
views
Is there a way to add impending state changes to a queue on a node server, and is it necessary?
I'm writing an express/socket.io-powered game server for a web game. I have a central map of game state objects, each representing an ongoing match, like so:
// map of gameId -> game (primary ...
- 1,124
1
vote
1
answer
1k
views
Pattern to limit message queue processing based on external API throttling
I have a project where there's an external API which implements throttling. Roughly speaking, I'm allowed to perform N requests per minute. I also have a message queue (Apache Kafka) whose consumers ...
- 167
-2
votes
1
answer
122
views
What is the proper way to define a generic cron-job
Am going to handle a CRON job module which will be a generic module, on the job execution time shall I push the events to a queue so let the respective module will pick and do the respective operation....
6
votes
2
answers
807
views
Design question for handling large volumes of messages in multi-tenant queue
I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario.
Producer produces events based ...
- 169
0
votes
1
answer
125
views
Distributed processing of a large number of queues
I have a rather basic application hosted on Kubernetes, which connects to a Mongo database.
The app has a wallet feature. A user can put money in their wallet using real-world payments (e.g. via ...
- 407