airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Issue with Airflow - Jobs are not triggered to run after a random period of time
Date Thu, 01 Sep 2016 14:18:33 GMT

Can you please create a Jira and add (debug) logs at the moment this happens, including Celery
logs please. I am trying to track this down. It seems only to happen with a CeleryExecutor.


Thanks
Bolke

> Op 1 sep. 2016, om 14:09 heeft Charalampos Paravalos <babis@rais.io> het volgende
geschreven:
> 
> Hi,
> 
> I am writting to ask for advise in an issue that I have with airflow and
> til now I have not managed to resolve. Wondering if someone else had
> something similar in the past.
> 
> So, we use airflow to schedule DAGs that will run some jobs periodically
> (every 30min/1hr). Jobs run as normal etc., but there are some times that
> suddenly after DAGs are finished, the next scheduled jobs do not start at
> all. It seems like the server does not kick off the scheduled jobs at all,
> for any of the DAGs defined (so no jobs are running on our server). When
> that happens I have to restart the scheduler so jobs are kicked on
> automatically after restart. And the jobs run until this issue appears
> again (I noticed it happening every 1 or 2 days, it is quite often).
> 
> This is very strange, tried to upgrade to 1.7.1.3 version but still that
> issue is here. We use 32 concurrent jobs with celery workers, the server is
> able to manage the load well.
> 
> I believe it has to do with the scheduler, but can't understand why.
> Backfilled jobs maybe? Can this be?
> 
> I am looking forward to hearing back from someone that has any ideas.
> Please let me know what information you might need about my setup anytime.
> 
> Thanks for your help!
> 
> Regards,
> Babis


Mime
View raw message