airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Micaheli <>
Subject Using multi processing inside a DAG
Date Wed, 28 Apr 2021 19:25:08 GMT

I would love your help with an issue I have.

Since multi threading isn't great with Python, I would like to create a
PythonOperator that utilizes multi processing inside it, in a Producer
Consumers design with a Queue.

Is that even possible inside a DAG?
Will that be supported with an Airflow DAG without causing unwanted
Does anyone have any experience creating such a DAG?

Thank you all,

View raw message