airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay Ramesh <vi...@change.org>
Subject Re: Scheduler silently dies
Date Fri, 24 Mar 2017 23:04:31 GMT
We literally have a cron job that restarts the scheduler every 30 min. Num
runs didn't work consistently in rc4, sometimes it would restart itself and
sometimes we'd end up with a few zombie scheduler processes and things
would get stuck. Also running locally, without celery.

On Mar 24, 2017 16:02, <lrohde@quartethealth.com> wrote:

> We have max runs set and still hit this. Our solution is dumber:
> monitoring log output, and kill the scheduler if it stops emitting. Works
> like a charm.
>
> > On Mar 24, 2017, at 5:50 PM, F. Hakan Koklu <fhakan.koklu@gmail.com>
> wrote:
> >
> > Some solutions to this problem is restarting the scheduler frequently or
> > some sort of monitoring on the scheduler. We have set up a dag that pings
> > cronitor <https://cronitor.io/> (a dead man's snitch type of service)
> every
> > 10 minutes and the snitch pages you when the scheduler dies and does not
> > send a ping to it.
> >
> > On Fri, Mar 24, 2017 at 1:49 PM, Andrew Phillips <aphillips@qrmedia.com>
> > wrote:
> >
> >> We use celery and run into it from time to time.
> >>>
> >>
> >> Bang goes my theory ;-) At least, assuming it's the same underlying
> >> cause...
> >>
> >> Regards
> >>
> >> ap
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message