airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Micaheli <c...@seadata.co.il>
Subject Using multi processing inside a DAG
Date Wed, 28 Apr 2021 19:25:08 GMT
Hello,

I would love your help with an issue I have.

Since multi threading isn't great with Python, I would like to create a
PythonOperator that utilizes multi processing inside it, in a Producer
Consumers design with a Queue.

Is that even possible inside a DAG?
Will that be supported with an Airflow DAG without causing unwanted
behaviors?
Does anyone have any experience creating such a DAG?

Thank you all,
Chen.

Mime
View raw message