airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Lin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-56) Airflow's scheduler can "lose" queued tasks
Date Mon, 09 May 2016 19:03:13 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276824#comment-15276824
] 

Kevin Lin commented on AIRFLOW-56:
----------------------------------

Wondering if this issue is only present in 1.7.0.
We've observed a very similar issue (queued tasks cannot be scheduled and run) in 1.6.2 with
Celeryd worker enabled. Thanks!

> Airflow's scheduler can "lose" queued tasks
> -------------------------------------------
>
>                 Key: AIRFLOW-56
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-56
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: executor, scheduler
>    Affects Versions: Airflow 1.7.0
>            Reporter: Bolke de Bruin
>            Priority: Blocker
>             Fix For: Airflow 1.7.1
>
>         Attachments: dag-queue.py
>
>
> Tasks that get into queued status can get "lost" on the scheduler/executor. They remain
in the DB with state "queued", but the scheduler won't pick them up as it will not look at
the DB currently. The executor does not see those tasks, although the scheduler has NOT been
restarted (ie. to see this issue it is not required to have "-n x" as a parameter to the scheduler).
> This happens *also* to tasks scheduled by the scheduler and not just with backfills created
by the SubDagOperator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message