airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <>
Subject Re: Airflow Dynamic tasks
Date Thu, 15 Aug 2019 18:52:28 GMT
What is your dynamic DAG doing? How long does it take to execute it just as
a python script (`time python`)?

As an Airflow admin, people may want to lower the DAG parsing timeout
configuration key to force people to not do crazy thing in DAG module
scope. At some point at Airbnb we had someone running a Hive query in DAG
scope, clearly that needs to be prevented.

Loading DAGs by calling a database can bring all sorts of surprises that
can drive everyone crazy. As mentioned in a recent post, repo-contained,
deterministic "less dynamic" DAGs are great, because they are
self-contained and allow you to use source-control properly (revert a bad
change for instance). That may mean having a process or script that
compiles external things that are dynamic into things like yaml files
checked into the code repo. Things as simple as parsing duration become
more predictable (network latency and database load are not part of that
equation), but more importantly, all changes become tracked in the code

yaml parsing in python can be pretty slow too, and there are solutions /
alternatives there. Hocon is great. Also C-accelerated yaml is possible:


On Wed, Aug 14, 2019 at 9:56 PM Bacal, Eugene <>

> Hello Airflow team,
> Please advise if you can. In our environment, we have noticed that dynamic
> tasks place quite of stress on scheduler, webserver and increase MySQL DB
> connections.
> We are run about 1000 Dynamic Tasks every 30 min and parsing time
> increases from 5 to 65 sec with Runtime from 2sec to 350+ . This happens at
> execution time then it drops to normal while still executing tasks.
> Webserver hangs for few minutes.
> Airflow 1.10.1.
> Example:
> Dynamic Tasks:
> Number of DAGs: 44
> Total task number: 950
> DagBag parsing time: 65.879642000000001
> Static Tasks:
> Number of DAGs: 73
> Total task number: 1351
> DagBag parsing time: 1.731088
> Is this something you aware of? Any advises on Dynamic tasks
> optimization/best practices?
> Thank you in advance,
> Eugene

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message