116 questions
76
votes
3
answers
107k
views
How to control the parallelism or concurrency of an Airflow installation?
In some of my Apache Airflow installations, DAGs or tasks that are scheduled to run do not run even when the scheduler doesn't appear to be fully loaded. How can I increase the number of DAGs or tasks ...
- 92k
10
votes
1
answer
3k
views
DAGs not clickable on Google Cloud Composer webserver, but working fine on a local Airflow
I'm using Google Cloud Composer (managed Airflow on Google Cloud Platform) with image version composer-0.5.3-airflow-1.9.0 and Python 2.7, and I'm facing a weird issue : after importing my DAGs, they ...
- 11.3k
11
votes
6
answers
10k
views
Google Cloud Composer and Google Cloud SQL
What ways do we have available to connect to a Google Cloud SQL (MySQL) instance from the newly introduced Google Cloud Composer? The intention is to get data from a Cloud SQL instance into BigQuery (...
- 156
9
votes
5
answers
10k
views
Broken DAG: (...) No module named docker
I have BigQuery connectors all running, but I have some existing scripts in Docker containers I wish to schedule on Cloud Composer instead of App Engine Flexible.
I have the below script that seems ...
- 2,651
7
votes
1
answer
4k
views
Airflow error importing DAG using plugin - Relationships can only be set between Operators
I have written an airflow plugin that simply contains one custom operator (to support CMEK in BigQuery). I can create a simple DAG with a single task that uses this operator and that executes fine.
...
- 95
6
votes
3
answers
8k
views
How to invoke a cloud function from google cloud composer?
For a requirement I want to call/invoke a cloud function from inside a cloud composer pipeline but I cant find much info on it, I tried using SimpleHTTP airflow operator but I get this error:
[2021-09-...
- 227
4
votes
2
answers
8k
views
Trigger Cloud Composer DAG with a Pub/Sub message
I am trying to create a Cloud Composer DAG to be triggered via a Pub/Sub message.
There is the following example from Google which triggers a DAG every time a change occurs in a Cloud Storage bucket:
...
- 117
2
votes
2
answers
5k
views
Google Cloud Composer BigQuery Operator- Get Jobs API HTTPError 404
I am trying to run a BigQueryOperator on GCC.
I have already succeeded in running for BigQueryCreateEmptyTableOperator and BigQueryTableDeleteOperator.
Here is my code for the dag:
import datetime
...
1
vote
1
answer
318
views
How can I dynamically change a bucket name in an Airflow DAG?
I have been using Airflow successfully to read a SQL file from a bucket and pass it to a BigQueryInsertJobOperator. Here is a minimal example to show my currently working setup:
with DAG(
"...
- 596
28
votes
4
answers
26k
views
Using Dataflow vs. Cloud Composer [closed]
I'd like to get some clarification on whether Cloud Dataflow or Cloud Composer is the right tool for the job, and I wasn't clear from the Google Documentation.
Currently, I'm using Cloud Dataflow to ...
user10503628
8
votes
1
answer
5k
views
Running docker operator from Google Cloud Composer
As for the documentation, Google Cloud Composer airflow worker nodes are served from a dedicated kubernetes cluster:
I have a Docker contained ETL step that I would like to run using airflow, ...
- 30.3k
8
votes
2
answers
4k
views
Google Cloud Composer(Airflow) - dataflow job inside a DAG executes successfully, but the DAG fails
My DAG looks like this
default_args = {
'start_date': airflow.utils.dates.days_ago(0),
'retries': 0,
'dataflow_default_options': {
'project': 'test',
'tempLocation': 'gs://...
- 1,944
7
votes
2
answers
9k
views
Cloud Composer (Airflow) jobs stuck
My Cloud Composer managed Airflow got stuck for hours since I've canceled a Task Instance that was taking too long (Let's call it Task A)
I've cleared all the DAG Runs and task instances, but there ...
- 1,676
7
votes
2
answers
8k
views
Can you get a static external IP address for Google Cloud Composer / Airflow?
I know how to assign a static external IP address to a Compute Engine, but can this be done with Google Cloud Composer (Airflow)? I'd imagine most companies need that functionality since they'd ...
5
votes
1
answer
10k
views
How do I read a file in the airflow cloud composer bucket?
To separate bigquery queries from the actual code I want to store the sql in a separate file and then read it from the python code. I have tried to add the file in the same bucket as the DAGs and also ...
- 23.5k