airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Issue with Airflow - Jobs are not triggered to run after a random period of time
Date Mon, 05 Sep 2016 06:07:21 GMT
Can you please specify more details of your setup? 

- Airflow version
- Celery version
- Kombu version
- Rabbitmq version

- Airflow config
- Rabbitmq config

Thanks!
Bolke

> Op 1 sep. 2016, om 22:55 heeft Sergei Iakhnin <llevar@gmail.com> het volgende geschreven:
> 
> Hi Bolke,
> 
> I'm now sure who you're directing the question to. In my case it does
> happen on celery. This was my original report
> 
> https://groups.google.com/forum/#!topic/airbnb_airflow/KrB9pp5ou3c
> 
> On Thu, Sep 1, 2016 at 10:35 PM Bolke de Bruin <bdbruin@gmail.com> wrote:
> 
>> Can you confirm that this happens on celery?
>> 
>> It awfully sounds like this:
>> http://stackoverflow.com/questions/27737990/django-celery-queue-getting-stuck
>> 
>> 
>> 
>> Sent from my iPhone
>> 
>>> On 1 sep. 2016, at 21:59, Sergei Iakhnin <llevar@gmail.com> wrote:
>>> 
>>> Alexandre talked about this being a known issue at least as far back as
>> 10
>>> months ago.
>>> 
>>>> On Thu, 1 Sep 2016, 21:46 Bolke de Bruin, <bdbruin@gmail.com> wrote:
>>>> 
>>>> Again please create a jira and add as much info as possible. Including
>>>> debug logs, executor logs, broker logs. If possible database dump.
>>>> 
>>>> Note airflow version, celery version, rabbitmq/redis etc. provide config
>>>> details.
>>>> 
>>>> We really need more info to hint this down as it has been quite elusive.
>>>> And I/we have not been able to replicate it.
>>>> 
>>>> Bolke
>>>> 
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On 1 sep. 2016, at 20:45, hilaviz@gmail.com wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> We face exactly the same issue...
>>>>> I tried to describe it here this week,
>>>>> But no one had a solution.
>>>>> 
>>>>> ‫ב-1 בספט׳ 2016, בשעה 17:54, ‏‏Sergei Iakhnin ‏<llevar@gmail.com>
>>>> כתב/ה:‬
>>>>> 
>>>>>> As far as I know even Airbnb themselves restart their schedulers
every
>>>> 30
>>>>>> minutes because of this issue. I ended up doing it as well with a
cron
>>>> job
>>>>>> after giving up hope that it would be fixed in the short term.
>>>>>> 
>>>>>>> On Thu, 1 Sep 2016, 16:03 Charalampos Paravalos, <babis@rais.io>
>>>> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am writting to ask for advise in an issue that I have with
airflow
>>>> and
>>>>>>> til now I have not managed to resolve. Wondering if someone else
had
>>>>>>> something similar in the past.
>>>>>>> 
>>>>>>> So, we use airflow to schedule DAGs that will run some jobs
>>>> periodically
>>>>>>> (every 30min/1hr). Jobs run as normal etc., but there are some
times
>>>> that
>>>>>>> suddenly after DAGs are finished, the next scheduled jobs do
not
>> start
>>>> at
>>>>>>> all. It seems like the server does not kick off the scheduled
jobs at
>>>> all,
>>>>>>> for any of the DAGs defined (so no jobs are running on our server).
>>>> When
>>>>>>> that happens I have to restart the scheduler so jobs are kicked
on
>>>>>>> automatically after restart. And the jobs run until this issue
>> appears
>>>>>>> again (I noticed it happening every 1 or 2 days, it is quite
often).
>>>>>>> 
>>>>>>> This is very strange, tried to upgrade to 1.7.1.3 version but
still
>>>> that
>>>>>>> issue is here. We use 32 concurrent jobs with celery workers,
the
>>>> server is
>>>>>>> able to manage the load well.
>>>>>>> 
>>>>>>> I believe it has to do with the scheduler, but can't understand
why.
>>>>>>> Backfilled jobs maybe? Can this be?
>>>>>>> 
>>>>>>> I am looking forward to hearing back from someone that has any
ideas.
>>>>>>> Please let me know what information you might need about my setup
>>>> anytime.
>>>>>>> 
>>>>>>> Thanks for your help!
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Babis
>>>>>> --
>>>>>> 
>>>>>> Sergei
>>> --
>>> 
>>> Sergei
>> 
> -- 
> 
> Sergei


Mime
View raw message