airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From harish singh <harish.sing...@gmail.com>
Subject airflow (1.7.0): cpu utilization reaches 70% and above
Date Sun, 12 Jun 2016 20:24:22 GMT
Hi guys,

We are running airflow (for about 3 months now) inside a docker container
on aws.

I just did a docker stats to check whats going on. The cpu consumption is
huge.
We have around 15 DAGS. Only one DAG is turned ON. the remaining are OFF.
The DAG runs with a HOURLY schedule.

Right now, airflow is consuming almost 1 complete core.
It seems there is some unnecessary spinning?
This doesnt look like the right behavior.
Is there a bug already filed for this? Or am not sure if there is something
incorrect in the way I am using the airflow configuration.

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %
      NET I/O               BLOCK I/O
CCC                         68.17%              619.7 MB / 2.147 GB
28.85%              1.408 GB / 939.4 MB   7.856 MB / 0 B
XXX                          64.36%              619.4 MB / 2.147 GB
28.84%              1.211 GB / 807.6 MB   7.856 MB / 0 B


Ariflow version 1.7.0

Airflow config:

sql_alchemy_pool_size = 5
sql_alchemy_pool_recycle = 3600
parallelism = 8
dag_concurrency = 8
max_active_runs_per_dag = 8



Thanks,

Harish

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message