airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Standish <dpstand...@gmail.com>
Subject Re: Airflow Dynamic Dags
Date Wed, 14 Aug 2019 23:51:21 GMT
If your dag generation process makes external calls over network, one
avenue of optimization is to cache your config (i.e. the thing that drives
your dag generation process) in local file (perhaps sqlite) and update that
file only periodically.  Or, just convert to fixed config object (e.g. a
dictionary in source control) and periodically compare with external source
to find deviation.


On Wed, Aug 14, 2019 at 4:40 PM Bacal, Eugene <ebacal@paypal.com.invalid>
wrote:

> Hello there,
>
> Please advise if you can. In our environment, we have noticed that dynamic
> tasks place a lot of stress on scheduler, webserver and DB (connections
> wise)
> We are running about 1000 Dynamic Tasks every 30 min and parsing time
> increases from 5 to 65 sec with Runtime from 2sec to 350+ . This happens at
> execution time then it drops to normal while still executing tasks.
> Webserver hangs for few minutes.
> Running 1.10.1.
>
> Dynamic Tasks:
> Number of DAGs: 44
> Total task number: 950
> DagBag parsing time: 65.879642000000001
>
>
> Static Tasks:
> Number of DAGs: 73
> Total task number: 1351
> DagBag parsing time: 1.731088
>
> Is this something you aware of? Any advises on Dynamic tasks
> optimization/best practices?
>
> Thank you in advance,
> Eugene
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message