airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: airflow (1.7.0): cpu utilization reaches 70% and above
Date Mon, 13 Jun 2016 15:14:13 GMT
The scheduler constantly attempts to schedule tasks, interacting with the
database and reloading DAG definition. In most larg-ish environments,
burning up to a CPU to run the scheduler doesn't seem outrageous to me.

If you want to reduce the CPU load related to the scheduler check out
SCHEDULER_HEARTBEAT_SEC and MAX_THREADS in the scheduler section of
`airflow.cfg`

Max

On Sun, Jun 12, 2016 at 1:24 PM, harish singh <harish.singh22@gmail.com>
wrote:

> Hi guys,
>
> We are running airflow (for about 3 months now) inside a docker container
> on aws.
>
> I just did a docker stats to check whats going on. The cpu consumption is
> huge.
> We have around 15 DAGS. Only one DAG is turned ON. the remaining are OFF.
> The DAG runs with a HOURLY schedule.
>
> Right now, airflow is consuming almost 1 complete core.
> It seems there is some unnecessary spinning?
> This doesnt look like the right behavior.
> Is there a bug already filed for this? Or am not sure if there is something
> incorrect in the way I am using the airflow configuration.
>
> CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %
>       NET I/O               BLOCK I/O
> CCC                         68.17%              619.7 MB / 2.147 GB
> 28.85%              1.408 GB / 939.4 MB   7.856 MB / 0 B
> XXX                          64.36%              619.4 MB / 2.147 GB
> 28.84%              1.211 GB / 807.6 MB   7.856 MB / 0 B
>
>
> Ariflow version 1.7.0
>
> Airflow config:
>
> sql_alchemy_pool_size = 5
> sql_alchemy_pool_recycle = 3600
> parallelism = 8
> dag_concurrency = 8
> max_active_runs_per_dag = 8
>
>
>
> Thanks,
>
> Harish
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message