All Questions
7 questions
1
vote
1
answer
51
views
Using Composer version 2.6.2 and Airflow version 2.5.3, getting AttributeError: 'bytes' object has no attribute 'encode' error in pubsub message [duplicate]
In logs can see the following error and DAG gets failed:-
[2024-03-27, 06:25:37 UTC] {logging_mixin.py:137} INFO - data:b'DNB-MS-MY,2024-03-26,Location' [2024-03-27, 06:25:37 UTC] {taskinstance.py:...
- 31
0
votes
1
answer
728
views
AirflowException python_callable param must be callable
I am getting error in Composer DAG while passing parameters to composer DAG.
AirflowException python_callable param must be callable
These parameters will be used in other python script (py_script....
- 71
2
votes
2
answers
3k
views
How to run independent requirements.txt file in apache airflow DAG
I have multiple aiflow dags running on composer environment. For now I haven't get a chance to install any extra python packages apart from the packages given by airflow. But now I want to install ...
1
vote
0
answers
489
views
BeamRunPythonPipelineOperator gives module not found error
I am working with creating a python BEAM pipeline with Dataflow in Cloud Composer 2.0. I am using BeamRunPythonPipelineOperator for the same. Below is the code
setup_script = path.join(configuration....
- 515
6
votes
3
answers
7k
views
FERNET_KEY configuration is missing when creating a new environment with the same DAGS
I'm using Composer (Airflow) in Google Cloud. I want to create a new environment and take my same DAGs and Variables from the old environment into the new one.
To accomplish this I do the following:
...
- 1,624
2
votes
2
answers
14k
views
Airflow: How can I access {{ execution date }} on my custom operator?
I built a custom operator on Airflow that calls an API to fetch data and then write it to BigQuery. However, the issue is that I have to pass the execution_date macro as an API param to be able to ...
- 31
0
votes
1
answer
3k
views
How to read dynamic argument airflow operator?
I am new in python and airflow dag.
I am following below link and code which is mention in answer section.
How to pass dynamic arguments Airflow operator?
I am facing issue to reading ...
- 539