airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nadeem Ahmed Nazeer <naz...@neon-lab.com>
Subject Re: Issue with Airflow - Jobs are not triggered to run after a random period of time
Date Thu, 01 Sep 2016 19:25:48 GMT
We too face the same issue with scheduler (Celery executor). I opened a
JIRA https://issues.apache.org/jira/browse/AIRFLOW-401 for this. You could
try to verify it against master as suggested by Bolke in my JIRA.

The workaround that works for us is to restart the scheduler every X mins
through an external script. This keeps the scheduler going hence launching
tasks without fail.

Thanks,
Nadeem

On Thu, Sep 1, 2016 at 11:45 AM, <hilaviz@gmail.com> wrote:

>
>
> We face exactly the same issue...
> I tried to describe it here this week,
> But no one had a solution.
>
> ‫ב-1 בספט׳ 2016, בשעה 17:54, ‏‏Sergei Iakhnin ‏<llevar@gmail.com>
כתב/ה:‬
>
> > As far as I know even Airbnb themselves restart their schedulers every 30
> > minutes because of this issue. I ended up doing it as well with a cron
> job
> > after giving up hope that it would be fixed in the short term.
> >
> >> On Thu, 1 Sep 2016, 16:03 Charalampos Paravalos, <babis@rais.io> wrote:
> >>
> >> Hi,
> >>
> >> I am writting to ask for advise in an issue that I have with airflow and
> >> til now I have not managed to resolve. Wondering if someone else had
> >> something similar in the past.
> >>
> >> So, we use airflow to schedule DAGs that will run some jobs periodically
> >> (every 30min/1hr). Jobs run as normal etc., but there are some times
> that
> >> suddenly after DAGs are finished, the next scheduled jobs do not start
> at
> >> all. It seems like the server does not kick off the scheduled jobs at
> all,
> >> for any of the DAGs defined (so no jobs are running on our server). When
> >> that happens I have to restart the scheduler so jobs are kicked on
> >> automatically after restart. And the jobs run until this issue appears
> >> again (I noticed it happening every 1 or 2 days, it is quite often).
> >>
> >> This is very strange, tried to upgrade to 1.7.1.3 version but still that
> >> issue is here. We use 32 concurrent jobs with celery workers, the
> server is
> >> able to manage the load well.
> >>
> >> I believe it has to do with the scheduler, but can't understand why.
> >> Backfilled jobs maybe? Can this be?
> >>
> >> I am looking forward to hearing back from someone that has any ideas.
> >> Please let me know what information you might need about my setup
> anytime.
> >>
> >> Thanks for your help!
> >>
> >> Regards,
> >> Babis
> > --
> >
> > Sergei
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message