airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Question about airflow scheduler
Date Mon, 20 Jun 2016 22:44:54 GMT
Hey Jason,

We run our scheduler without setting this. It will cause the scheduler to
kill itself ever N loops. If you use it in conjunction with a process
monitor (e.g. supervisord), you can have the scheduler kill itself, and
then the process monitor will restart it. This gives you short lived
schedulers that clear their state ever N loops.

TBH, I don't think that it's required anymore. It seems like a hack for
when the scheduler was more stateful, and needed to restart itself
periodically to clear its in-memory state.

Cheers,
Chris

On Sun, Jun 19, 2016 at 2:59 PM, Jason Chen <chingchien.chen@gmail.com>
wrote:

> Hi Airflow team,
>
> To run "airflow scheduler", you can specify an argument "SCHEDULER_RUNS"
> Something like "airflow scheduler -n ${SCHEDULER_RUNS}".
>
> What it's used for ?
> Side effects when setting it to un-limit ?
>
> Thanks.
>
> Jason
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message