airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siddharth anand <san...@apache.org>
Subject Re: execution_timeout ignored for PythonOperator in 1.7.1.2?
Date Tue, 19 Jul 2016 02:24:10 GMT
Alyssa,
Are you running UTC everywhere? That is still an unfortunate requirement in
some parts of airflow. I only ask because this is a common stumbling point
-- one I have debugged for other before.

You need to run your DB and your airflow servers in UTC.

-s

On Thu, Jun 9, 2016 at 9:03 AM, Alyssa Kwan <alyssa.kwan@cloverhealth.com>
wrote:

> We're not using LocalExecutor. It's CeleryExecutor. We're running on
> virtual hardware and perhaps the signal module is simply slow. (It
> shouldn't be this slow.) We *do* get a timeout eventually. Just not at the
> 10 minute mark. It doesn't trip until 5 hours.
>
>
> On 8 Jun 2016, at 8:22, Maxime Beauchemin wrote:
>
> I created this JIRA to document the issue.
>> https://issues.apache.org/jira/browse/AIRFLOW-221
>>
>> Max
>>
>> On Wed, Jun 8, 2016 at 8:16 AM, Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>> Are you using the LocalExecutor? there's a limitation around using the
>>> task timeout in LocalExecutor under some circumstances. If that's the
>>> case
>>> you should see this somewhere in your logs:
>>>
>>>
>>> https://github.com/apache/incubator-airflow/blob/master/airflow/utils/timeout.py#L45
>>>
>>> We may want to move the logic enforcing the timeout from the context
>>> manager/signal approach to BaseJob.
>>>
>>> Max
>>>
>>> On Tue, Jun 7, 2016 at 5:47 PM, Alyssa Kwan <
>>> alyssa.kwan@cloverhealth.com>
>>> wrote:
>>>
>>> Hi,
>>>>
>>>> We're having trouble with a `PythonOperator`'s `execution_timeout`. It's
>>>> being set directly in the constructor as
>>>> `datetime.timedelta(minutes=10)`.
>>>> However, the task doesn't timeout until almost 5 hours is passed. This
>>>> behavior is pretty consistent. It's merely a SQL query issued by the
>>>> Python
>>>> callable that can get deadlocked.
>>>>
>>>> Any thoughts?
>>>>
>>>> Thanks,
>>>> Alyssa
>>>>
>>>
>>>
>>>
>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message