airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Imberman (Jira)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-6175) Executor changes queued tasks to scheduled without removing from task queue
Date Thu, 05 Dec 2019 00:00:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Imberman updated AIRFLOW-6175:
-------------------------------------
    Description: 
There is a bug caused by scheduler_jobs refactor which leads to task failure and scheduler
locking.

Essentially when a there is an overflow of tasks going into the scheduler, the tasks are set
back to scheduled, but are not removed from the executor's queued_tasks queue.

This means that the executor will attempt to run tasks that are in the scheduled state, but
those tasks will fail dependency checks. Eventually the queue is filled with scheduled tasks,
and the scheduler can no longer run.

> Executor changes queued tasks to scheduled without removing from task queue
> ---------------------------------------------------------------------------
>
>                 Key: AIRFLOW-6175
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6175
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 1.10.6
>            Reporter: Daniel Imberman
>            Priority: Blocker
>
> There is a bug caused by scheduler_jobs refactor which leads to task failure and scheduler
locking.
> Essentially when a there is an overflow of tasks going into the scheduler, the tasks
are set back to scheduled, but are not removed from the executor's queued_tasks queue.
> This means that the executor will attempt to run tasks that are in the scheduled state,
but those tasks will fail dependency checks. Eventually the queue is filled with scheduled
tasks, and the scheduler can no longer run.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message