airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adinata <mail.die...@gmail.com>
Subject Re: Scheduler defunct Long Running Task with LocalExecutor
Date Mon, 15 Aug 2016 07:00:18 GMT
Hi Nadeem,

I think my scheduler stuck because there is a remaining running task in the
executor. I guess that's why the scheduler `defunct`-ed. It won't do to
restart the scheduler because it will kill the running task. I'm not using
CeleryExecutor, and I'm not planning on trying it.

@all
If I wasn't wrong, in master we don't need to do `n numruns` of scheduler.
Does it safe to run master in production? Any guide on how to install the
master version?

If you don't recommend to run master in production, is there any trade off
in running multiple scheduler instance (e.g., under supervisor) with single
parallelism configuration?

Thanks,

--
*Adinata*
TOKI 2009
SMAN Plus Provinsi Riau 9th
13509022 - Informatika ITB 2009
Engineer - UrbanIndo.com

On Mon, Aug 15, 2016 at 11:23 AM, Nadeem Ahmed Nazeer <nazeer@neon-lab.com>
wrote:

> Hi Adinata,
>
> I had the same issue and have logged a JIRA for this (
> https://issues.apache.org/jira/browse/AIRFLOW-401).
>
> Bolke has commented on JIRA to have the issue verified against master
> branch and it is on my to-do list. Looks like there were a lot of scheduler
> fixes made since release. I haven't verified this on master yet, if you get
> a chance to verify, please let us know what you find.
>
> The only workaround I found is to set up a process that restarts the
> scheduler every X minutes although it might not be clean. Also, did you try
> with Celery Executor?
>
> Thanks,
> Nadeem
>
> On Sat, Aug 13, 2016 at 10:38 PM, Adinata <mail.dieend@gmail.com> wrote:
>
> > Hi all,
> >
> > I think I have problem with long running task in LocalExecutor, that make
> > other scheduler process become defunct.
> > I'm using airflow 1.7.1.3, and running airflow scheduler -n 20. When it's
> > on the defunct state, it's unable to schedule any tasks. Do you have any
> > possible solution?
> >
> > Attached the screenshot in this email.
> >
> > --
> > *Adinata*
> > Engineer - UrbanIndo.com
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message