airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergei Iakhnin <lle...@gmail.com>
Subject Re: Issue with Airflow - Jobs are not triggered to run after a random period of time
Date Thu, 01 Sep 2016 14:54:10 GMT
As far as I know even Airbnb themselves restart their schedulers every 30
minutes because of this issue. I ended up doing it as well with a cron job
after giving up hope that it would be fixed in the short term.

On Thu, 1 Sep 2016, 16:03 Charalampos Paravalos, <babis@rais.io> wrote:

> Hi,
>
> I am writting to ask for advise in an issue that I have with airflow and
> til now I have not managed to resolve. Wondering if someone else had
> something similar in the past.
>
> So, we use airflow to schedule DAGs that will run some jobs periodically
> (every 30min/1hr). Jobs run as normal etc., but there are some times that
> suddenly after DAGs are finished, the next scheduled jobs do not start at
> all. It seems like the server does not kick off the scheduled jobs at all,
> for any of the DAGs defined (so no jobs are running on our server). When
> that happens I have to restart the scheduler so jobs are kicked on
> automatically after restart. And the jobs run until this issue appears
> again (I noticed it happening every 1 or 2 days, it is quite often).
>
> This is very strange, tried to upgrade to 1.7.1.3 version but still that
> issue is here. We use 32 concurrent jobs with celery workers, the server is
> able to manage the load well.
>
> I believe it has to do with the scheduler, but can't understand why.
> Backfilled jobs maybe? Can this be?
>
> I am looking forward to hearing back from someone that has any ideas.
> Please let me know what information you might need about my setup anytime.
>
> Thanks for your help!
>
> Regards,
> Babis
>
-- 

Sergei

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message