airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Guziel <alex.guz...@airbnb.com.INVALID>
Subject Re: Memory Issues with Airflow Subdag
Date Wed, 05 Apr 2017 23:55:52 GMT
Which container is using the memory?

On Wed, Apr 5, 2017 at 2:23 PM, Alex Keating <alex@wayup.com> wrote:

> Hey Everyone,
>
> We recently added a subdag with over 200 tasks on it and any time airflow
> is running the cpu and memory usage spikes taking down the server. I am
> putting the subdag in a factory outside of the dags folder and importing it
> into a dag. We are using the Celery Executor, and running the worker,
> webserver, scheduler, and redis on a t2.medium instance with everything in
> a separate docker container. Is this normal? Why might this subdag be
> spiking our memory and cpu?
>
> --
> Alexander Keating
>
> Software Engineer | WayUp <http://www.wayup.com>
>
> 646.535.8724 | alex@wayup.com
>
> We're hiring at WayUp!
> <https://www.wayup.com/joinus/?utm_source=email&utm_medium=
> opsignature&utm_campaign=joinus>
>
> We help students get hired. <http://bit.ly/sigsigsig>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message