airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wood stock <ws1...@gmail.com>
Subject Re: Specifying memory limit for task
Date Fri, 05 Aug 2016 14:57:33 GMT
I think you can define a new operator which take an existing operator base
class and keep monitoring the memory usage (maybe base on airflow
heartbeat?).

On Thu, Aug 4, 2016 at 8:23 PM, Adinata <mail.dieend@gmail.com> wrote:

> Is there any way for task instances to limit the memory task execution?
>
> I run the worker inside docker container, hence when it uses high memory,
> the container was killed.
> Airflow then detect it as zombie (well the email notification failure told
> me that). It would be great if
> I know it was killed because of oom, so I know what to fix after instead of
> wondering first why it was
> becoming a zombie.
>
> Thanks
> --
> *Adinata*
> Engineer - UrbanIndo.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message