airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: AIRFLOW-1258
Date Wed, 26 Jul 2017 19:25:50 GMT
Can you explain what is solved by the patch? It only seems to set a empty default value? 

If that solves it it also seems to be a bug with celery. Or did you set some options? That
would be great to share. 

Bolke 

Sent from my iPhone

> On 26 Jul 2017, at 19:44, Jawahar Panchal <jpanchal@hawk.iit.edu> wrote:
> 
> Howdy - wanted to provide an update - the following patch applied manually to a clean
1.8.1 install addressed the issue:
> 
> https://github.com/apache/incubator-airflow/pull/2143 <https://github.com/apache/incubator-airflow/pull/2143>
> 
> We have confirmed/verified this with jobs running over multiple hours on our instance
- so the above pull request can close both of the below:
> 
> https://issues.apache.org/jira/browse/AIRFLOW-966 <https://issues.apache.org/jira/browse/AIRFLOW-966>
> https://issues.apache.org/jira/browse/AIRFLOW-1258 <https://issues.apache.org/jira/browse/AIRFLOW-1258>
> 
> As I have seen further discussion on 1.8.2 with the final build not being tagged yet
(iirc), would it be possibly to merge this for the 1.8.2 release? As it currently is, someone
using Airflow+Celery will have a broken configuration for long-running jobs - would the dev
team consider this major/critical enough to include?
> 
> Cheers,
> J
> 
>> On Jul 17, 2017, at 3:15 PM, Jawahar Panchal <jpanchal@hawk.iit.edu> wrote:
>> 
>> Hi again!
>> 
>> 
>>> On Jul 16, 2017, at 3:22 AM, Alex Guziel <alex.guziel@airbnb.com.INVALID <mailto:alex.guziel@airbnb.com.invalid>>
wrote:
>>> 
>>> I think this may be related to a celery bug. I'll follow up with more
>>> details later.
>>> 
>> 
>> Just replying back to the note earlier in the thread - apologies for the earlier
top-posting, got a bit excited that I might have found the issue, and of course lack of sleep
results in one doing terrible, terrible things… :)
>> 
>> Any idea if my suspicion around the 1h default visibility timeout between celery/redis
is the culprit?
>> 
>>> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jpanchal@hawk.iit.edu>
>>> wrote:
>>> 
>>>> Hi!
>>>> 
>>>> I am currently running a couple of long-running tasks on a
>>>> database/dataset at school for a project that results in behavior/log
>>>> output similar to what was flagged in this bug:
>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>> 
>>>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>>>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>>>> change, needless to say any pointers from the dev team on the right
>>>> direction to look within the codebase would be much appreciated! :)
>>>> 
>>>> Thanks in advance for everyone’s/anyone's time and help - am not an
>>>> Airflow expert, but am hopefully learning quickly enough to help resolve
>>>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>>> 
>>>> Cheers,
>>>> J
>>>> 
>>>> 
>> 
>> Cheers,
>> J
> 

Mime
View raw message