airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: Specifying memory limit for task
Date Fri, 05 Aug 2016 23:07:05 GMT
We're adding cgroups semantics to Airflow's BaseOperator, along with
integration to activate those settings when firing up the task instance as
a subprocess. Paul Yang (@Airbnb) is working on a prototype / design doc
and should be able to share more shortly.

It lines up nicely with the recent work around unix impersonation as they
are all new optional BaseOperator params that wraps the subprocess. There
features will require Airflow to run with elevated privileges, which
Airflow will "downstep" from as it runs the tasks instances as subprocesses.

Max

On Fri, Aug 5, 2016 at 11:03 AM, Lance Norskog <lance.norskog@gmail.com>
wrote:

> Linux cgroups gives Docker the ability to control memory use inside a
> container.
>
> https://www.cloudsigma.com/manage-docker-resources-with-cgroups/
>
> Here is an example that allocates the CPU:
> http://blog.viktorpetersson.com/post/115562026784/using-
> cgroups-with-docker-on-ubuntu-1404
>
> (I have not worked with cgroups.)
>
> On Fri, Aug 5, 2016 at 7:57 AM, wood stock <ws1088@gmail.com> wrote:
>
> > I think you can define a new operator which take an existing operator
> base
> > class and keep monitoring the memory usage (maybe base on airflow
> > heartbeat?).
> >
> > On Thu, Aug 4, 2016 at 8:23 PM, Adinata <mail.dieend@gmail.com> wrote:
> >
> > > Is there any way for task instances to limit the memory task execution?
> > >
> > > I run the worker inside docker container, hence when it uses high
> memory,
> > > the container was killed.
> > > Airflow then detect it as zombie (well the email notification failure
> > told
> > > me that). It would be great if
> > > I know it was killed because of oom, so I know what to fix after
> instead
> > of
> > > wondering first why it was
> > > becoming a zombie.
> > >
> > > Thanks
> > > --
> > > *Adinata*
> > > Engineer - UrbanIndo.com
> > >
> >
>
>
>
> --
> Lance Norskog
> lance.norskog@gmail.com
> Redwood City, CA
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message