airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramandumcs@gmail.com"<ramandu...@gmail.com>
Subject Re: Benchmarking Airflow
Date Wed, 03 Jul 2019 04:33:21 GMT
Hi Sergio,

We did some benchmarking with Local & K8 Executor Mode. We observed that Each Airflow
Tasks takes ~100 MB of memory in Local Executor Mode.
With 16 GB of RAM we could run ~140 concurrent tasks. After this we started getting "can not
allocate memory error".
With K8 Executor memory footprint of task(worker Pod) increases to ~150 MB.
We also observed that  scheduling latency increases with increase in Number of DAG files.
Airflow.cfg's config "max_threads" controls the number of Dag files to be processed parellely
in every scheduling loop.
so 
Time to process DAG = ((Number of Dags)/max_threads) * (Scheduler Loop Time)

Thanks,
Raman Gupta
 


On 2019/07/02 22:55:26, Sergio Kef <sergiokef@gmail.com> wrote: 
> Hey folks,
> 
> Do we have something like airflow benchmarks?
> Seems that many people seem to struggle to understand the limitations of
> airflow  (me included).
> 
> Is there some existing work on bechmarking (ie define a few common cases
> and measure performance while increase volume of tasks)?
> 
> I know it's quite challenging task to compare the different executors or
> different versions, etc. but even if we start very simple (eg resources
> required for an idle airflow scheduler), I think we will start having
> useful insights.
> 
> What's your thoughts?
> S.
> 

Mime
View raw message