Questions tagged [apache-kafka]
Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.
99 questions
5
votes
3
answers
1k
views
How to design a distributed system with an event broker where strong ordering is required?
I was trying to model a fairly simple real world model inside a distributed system and got stuck thinking about timing and order and would appreciate some external view on it.
Assuming I have this ...
- 799
3
votes
1
answer
170
views
Deciding on granularity of SOLID's SRP
I am currently working in a project where we are utilizing kafka as a Message Queue. We have two use cases here, one is to consume the messages in parallel, no ordering of any kind required. And one ...
- 139
0
votes
1
answer
349
views
Best Practices for Designing Kafka Message Processing with Independent Tasks
I am designing a message processing system for Kafka topic A.
Each message has a message_type and needs to query DB to determine the tasks required for this message_type. The tasks are independent of ...
- 117
0
votes
0
answers
201
views
How does each application replica read a unique message from the message broker
In RabbitMQ streams or Kafka, messages are not deleted after being consumed. If you have a consumer application replicated across multiple Kubernetes pods, how can you ensure that each pod picks up a ...
- 9
-2
votes
1
answer
238
views
Transaction Management Across Microservices with Rollback Mechanism using Kafka
I am currently developing a microservices architecture consisting of two services, and I'm facing a challenge related to transaction management and error handling.
Microservice A: This service is ...
- 1
1
vote
1
answer
160
views
Designing a Kafka consumer that gets a high throughput of "redundant" events
Context:
We have 2 services - a user-facing API server and a background processing Kafka worker.
Both of these are hosted on AWS EKS, and we use AWS MSK for Kafka.
The job of the API server is to ...
- 111
3
votes
2
answers
455
views
In Hexagonal Architecture how do I implement a Dead-Letter Queue?
I have an application using Hexagonal Architecture. I want to implement a DLQ in 2 scenarios:
When the event streaming adapter's incoming message is unreadable (invalid payload, etc)
When the domain ...
- 133
0
votes
2
answers
172
views
What if the kafka leader crashes before committing the offset to zookeeper?
I'm not encountering any issue in prod or anything, I'm just reading about Kafka and I'm curious about this case.
As I understand,
A producer sends messages to the leader.
The replicas copy the ...
- 129
1
vote
1
answer
107
views
How to test integration with an infinite generator
I'm struggling to (integration) test a component that makes use of an infinite generator.
Background:
At a really high level, this component is essentially a "middleman" in a kafka workflow. ...
- 271
0
votes
1
answer
124
views
Data file ingestion with minio and kafka
I want to collect a lot of files (file data + metadata) from local servers to a central server. Files are important, need to ensure that no files are lost
Local servers: implement a collector to ...
- 117
0
votes
2
answers
245
views
Seeking a Third Opinion on Kafka Consumer Implementation and Architectural Disagreements
I've been at my current job for about 4-5 months, mainly working with Go, and I have no prior experience with Kafka. Before this, my background was in JavaScript, Node.js, React, etc. I recently got a ...
- 274
2
votes
2
answers
1k
views
How to effectively scale one microservice to multiple instances
Let’s say I have a booking microservice (single instance) with its own DB.
If I want to horizontally scale the service with multiple instances(exact same copy of the business logic code) running on ...
- 157
2
votes
2
answers
197
views
Using json objects (with maintained schema in registry) vs. string vs. hash as message key
Folks,
I am wondering if there is any dis/advantage to using json objects as message keys.
Assume I have 2 event streams coupled with 2 tables in a source system. Each of those tables have its own ...
- 31
1
vote
1
answer
170
views
How to approach converting to and from kafka messages?
I would like to have a series of small stand-alone services that would either consume a Kafka topic and output the data into a different system or the reverse: receive data from a system and produce ...
- 11
1
vote
1
answer
512
views
KRaft design architecture - Separation of Controllers and Brokers
We want to install Kafka Kraft cluster on 35 machines ( machines are DELL physical machines - type R760 And OS is Linux RHEL 8.6 version )
Since we are dealing with Kafka production cluster , we want ...
- 111