airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bacal, Eugene" <eba...@paypal.com.INVALID>
Subject Airflow Dynamic tasks
Date Thu, 15 Aug 2019 04:56:17 GMT
Hello Airflow team,

Please advise if you can. In our environment, we have noticed that dynamic tasks place quite
of stress on scheduler, webserver and increase MySQL DB connections.
We are run about 1000 Dynamic Tasks every 30 min and parsing time increases from 5 to 65 sec
with Runtime from 2sec to 350+ . This happens at execution time then it drops to normal while
still executing tasks.
Webserver hangs for few minutes.

Airflow 1.10.1.
MySQL DB

Example:

Dynamic Tasks:
Number of DAGs: 44
Total task number: 950
DagBag parsing time: 65.879642000000001

Static Tasks:
Number of DAGs: 73
Total task number: 1351
DagBag parsing time: 1.731088

Is this something you aware of? Any advises on Dynamic tasks optimization/best practices?

Thank you in advance,
Eugene


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message