All Questions
64 questions
0
votes
0
answers
39
views
Error 'Failed to import plugin opt/airflow/plugins/operatos/. . . ' when running docker-compose up for images
In my project, I build a docker image on my local machine, then pull the image to the production machine.
On my local machine, I can still run the image using the docker-compose up command to create a ...
- 1
0
votes
1
answer
158
views
DAG's from DAG Folder is not visible in Airflow Home DAG (localhost:8080)
I have 3 Dag files (example1.py, example2.py and example3.py) in DAG folder in airflow/airflow-docker/dags folder (docker container in vscode) and they're not showing up in the Airflow Web Home Page, ...
0
votes
0
answers
110
views
Docker, "The scheduler does not appear to be running" and "The DAGs list may not update, and new tasks will not be scheduled." in Airflow
I am new to Docker and just started learning it, so I have stumbled into this problem that I do not know how to solve.
The airflow page shows an empty list of dags. When I check the Docker desktop app ...
0
votes
0
answers
108
views
java.io.InvalidClassException: org.apache.spark.scheduler.Task; local class incompatible
I'm building an ETL for fetching data from an API, transforming it and uploading it in mongodb.
I use pyspark to distribute this process and I'm trying to use airflow to automate it.
However, I’ve hit ...
- 1
0
votes
0
answers
79
views
Airflow DAG Import Error Avro - installed everywhere
import avro.schema
ModuleNotFoundError: No module named 'avro'
I have this error when I try to import a DAG in Airflow.
I have avro installed in the docker image as shown below:
Installed /home/...
- 3
0
votes
0
answers
157
views
Docker Airflow - ModuleNotFoundError: No module named 'playwright'
I have installed Docker and Docker Compose, and I'm setting up Airflow on my VM using Docker. I followed this guide: Running Airflow on Google Compute Engine (GCE) from Scratch.
Everything was ...
0
votes
0
answers
224
views
Airflow BashOperator Fails to Find Script in Mounted Volume
I'm facing an issue while using the BashOperator in an Airflow to run a Python script. It is running inside docker container. The problem is that it can't find the path to the script, even though I've ...
-1
votes
1
answer
526
views
Connect to a MySQL Database running in a Docker container, from another container
I have a docker-compose.yml file in which I am running airflow, everything is running ok. But I have another docker-compose.yml file for a container in which I set up a MySQL database. I want to write ...
1
vote
1
answer
765
views
PermissionError: [Errno 13] Permission denied: '<project directory>'
I am troubleshooting using airflow for some MLops work (see this).
Here's the log error:
2c0cbc30645c
*** Found local files:
*** * /opt/airflow/logs/dag_id=ml_pipeline/run_id=manual__2023-07-12T17:...
- 301
1
vote
1
answer
421
views
Building airflow in docker compose with pycurl in requirements gets Permission denied: 'curl-config'
The problem
When building a docker image for apache airflow, build fails at RUN pip install -r requirements.txt (when installing pycurl package) with:
Permission denied: 'curl-config'
What I think ...
- 418
0
votes
1
answer
237
views
Airflow Python path trying to be read does not exist resulting in ImportError
I'm running Airflow in docker and all runs smoothly except when I want to create a new dag using airflow libraries. More specifically I'm importing
from airflow import DAG
from datetime import ...
- 1
1
vote
1
answer
893
views
Trouble Attempting to run a Docker Image from an Airflow DAG
I am trying to set up a DAG to run a Docker Image but having an issue running it.
I'm using the official Airflow docker-compose file.
My DAG file:
from airflow import DAG
from airflow.providers.docker....
- 135
0
votes
0
answers
284
views
subprocess.CalledProcessError: Command '('/usr/lib/R/bin/R', 'CMD', 'config', 'LIBnn')' returned non-zero exit status 127
I'm trying to create an automation of my ETL process using Apache Airflow, but I'm using a library in R that already extracts the data from the FTP server and returns them to me treated. For that I'm ...
1
vote
0
answers
730
views
Airflow localhost:8080 does not load (using docker-compose.yaml & celery executor)
I am trying to run airflow webserver at localhost:8080, but the URL never finishes loading and the airflow UI cannot be seen. My docker-compose.yaml looks like below.
Please note I am using a remote ...
0
votes
1
answer
3k
views
Apache Airflow Dag Import Error - FileNotFoundError: [Errno 2] No such file or directory
I have 2 tasks in my Dag. The 1st task pulls some data from an API, creates a directory at "/tmp/data" on the worker then writes the data from the API to a file called "test_data.json&...
- 1