All Questions
11 questions
1
vote
0
answers
93
views
AdminClient clientId=adminclient-1 Node 1 disconnected. Apache Kafka y Spark
I'm running a Docker Compose setup with Spark and Kafka, but I'm encountering connectivity issues when Spark tries to connect to Kafka. My docker-compose.yml file includes services for both Kafka and ...
0
votes
0
answers
25
views
Spark worker cannot connect to Kafka Broker [duplicate]
I setup a SparkCluster according to this instruction and I wanted to extend it's functionality by adding Kafka broker.
However I have problem with connecting Spark worker to Kafka broker
I get ...
0
votes
0
answers
71
views
Kafka partially connecting to cassandra to write streams of data
I am trying to write streams of data from kafka to cassandra on docker container automated with Airflow DAG. Here are the things that are working as expected.
DAG sending data and I can see the ...
- 1
1
vote
0
answers
101
views
Spark job stuck at running process
I use docker-compose to make a cluster of Spark and Kafka. When I tried to consume a topic by submitting it to this cluster, the job was always stuck and not executed.
Here is my docker-compose file:
...
0
votes
1
answer
789
views
How to connect Spark to Kafka when both are running on Docker? [duplicate]
I'm attempting to set up a quick POC on my Mac laptop (using Docker) to help demonstrate a streaming data ingestion flow using MySQL, Debezium, Kafka and Spark.
The MySQL / Debezium / Kafka ...
- 15.6k
0
votes
1
answer
608
views
How to install Kafka inside a docker image?
I am working on docker for the first time where I am trying to run my pyspark code on a docker container.
This my project structure.
My Dockerfile content:
from gcr.io/datamechanics/spark:platform-3....
- 2,123
4
votes
1
answer
3k
views
Connect PySpark to Kafka from Docker container
I have a Kafka cluster that I'm managing with Docker.
I have a container where I'm running the broker and another one where I run the pyspark program which is supposed to connect to the kafka topic ...
- 1,105
0
votes
1
answer
619
views
Jupyterlab container cannot connect to kafka localhost server on port 9092
I have docker-compose containing a kafka server, a zookeeper, spark and a jupyter-lab.
This is my code of jupyter-lab dockerfile
ARG debian_buster_image_tag=8-jre-slim
FROM openjdk:${...
-2
votes
1
answer
262
views
Whenever I am getting my spark job in dockercontainer,I am getting Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
The error I am getting is:-
20/11/02 13:34:51 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-366ac503-c5a4-4338-869c-84786983aab3--188679505-driver-0-1, groupId=spark-kafka-source-...
- 21
0
votes
1
answer
862
views
Spark in Docker container does not read Kafka input - Structured Streaming
When the Spark job is run locally without Docker via spark-submit everything works fine.
However, running on a docker container results in no output being generated.
To see if Kafka itself was working,...
1
vote
1
answer
2k
views
Using Docker and connectors to connect kafka to spark, spark to cassandra and kafka to cassandra
Detail: We have dockerized Kafka, Cassandra and Spark, using wurstmeister/kafka, strapdata/elassandra and bde2020/spark-master images in docker-compose.
What we want to do is to connect the ...
- 6,854