33,401 questions
0
votes
0
answers
32
views
FastAPI WebSocket + Kafka: Only the last connected client receives messages [closed]
I'm building a real-time dashboard using FastAPI WebSockets + Kafka. Everything works perfectly for a single user, but when multiple users connect, only the most recently connected user receives ...
Best practices
0
votes
2
replies
38
views
Kafka Topic Architecture for 200+ IoT Devices: One Topic per Device vs. Shared Topic?
I am designing a system using .NET that acts as a MES system between 200+ pieces of equipment (IoT devices). The flow is bidirectional:
Equipment -> My .NET System -> Equipment
I need low ...
0
votes
1
answer
38
views
Deploying Conduktor Console for Apache Kafka with Docker Compose FATAL: role "root" does not exist
I am trying to install an instance of Conduktor Console using a Docker image, following the instructions from the GitHub page:
Console Only: Start Conduktor and connect your own Kafka
# Start ...
0
votes
0
answers
20
views
Apache Pinot queries getting slower over time while consuming large real-time data — how to fix? [closed]
I’m using Apache Pinot and ingesting a large amount of real-time data. Everything works initially, but after some time my queries start slowing down significantly.
I’m new to Pinot and not sure what ...
0
votes
0
answers
65
views
Kafka producer not able to reconnect to pipeline on restart
System is on two site, facing issues on resiliency so using circuit breaker. When I am on close state then sending message to the primary site on call back on secondary site both are using different ...
1
vote
0
answers
35
views
Why does the triggering feature not support exceptions when using Kafka engine tables
I have subscribed to a Kafka cluster A with some Kafka engine tables before, and they are working normally. I have created a new Kafka cluster now, but after subscribing to the topics of the new ...
Best practices
0
votes
0
replies
30
views
Throw message on DLT/DLQ from GlobalKTable using Kafka Streams
I am reading data from a Kafka topic using Kafka Streams and more specifically GlobalKTable to populate my store. In the case of corrupt data that can not successfully be parsed I wish to throw a ...
Best practices
0
votes
7
replies
57
views
How to populate two stores from one topic using Kafka Streams GlobalKTable
I am using Kafka Streams, and trying to figure out the best practice for handling multiple stores that are populated using the same data without having to read it twice.
My Kafka Streams service is ...
1
vote
0
answers
58
views
Ack messages with StreamBridge::send and Kafka
Context. I have to use StreamBridge::send() with Kafka. This is a hard, external requirement. The call itself looks like streambridge.send(binding, message). All good.
To capture the non-immediate ...
Best practices
0
votes
1
replies
55
views
How to implement an automated E2E testing for an Event Driven Architecture?
I have an EVDA system that each component communicate to each other using Kafka and Outbox Pattern.
How to implement an automated E2E testing that I start producing an event from an upstream, then ...
2
votes
1
answer
28
views
Facing connectivity issue on this traditional compose file on podman, unbale to rectify what is exact issue
Image details:
Service
Container ID
Image
Status
Ports
Name
Zookeeper
17025a6f1e4b
confluentinc/cp-zookeeper:latest
Up 18s
2181,2888,3888
composefiles_zookeeper_1
Kafka
4a0964b3f368
confluentinc/cp-...
0
votes
1
answer
39
views
Is there an Kafka Connect SMT to inject an header to the event body/request
I am looking for a way to inject Kafka headers into a the message body when using AWS MSK connector.
Something similar to:
"transforms": "InsertField",
"transforms.InsertField....
0
votes
0
answers
29
views
Apache NIFI 2.5.0 KafkaPublisher can not publish message larger than 30 MB
Apache NIFI 2.5.0 KafkaPublisher can not publish message larger than 30 MB.
I have adjust the Max Request Size to 50MB but still in vain.
The topic on Kafka has config the message size to 50 MB.
I ...
Best practices
0
votes
0
replies
54
views
Spring Kafka consumer reprocessing messages after switching from AckMode.MANUAL_IMMEDIATE to MANUAL
I have two Spring Kafka consumers that process batches of messages with manual acknowledgements. This setup has been running correctly for a long time with AckMode.MANUAL_IMMEDIATE.
Recently, I ...
1
vote
0
answers
68
views
Oracle GoldenGate Kafka Connect handler — TimeoutException: Expiring records when producing to Kafka topic
I’m trying to integrate Oracle GoldenGate for Big Data (Kafka Connect handler) with Confluent Kafka running in Docker.
The handler starts successfully, trail files are read, and mapping works — but ...