airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1104) Concurrency check in scheduler should count queued tasks as well as running
Date Tue, 31 Jul 2018 20:26:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564331#comment-16564331
] 

ASF subversion and git services commented on AIRFLOW-1104:
----------------------------------------------------------

Commit ed972042a864cd010137190e0bbb1d25a9dcfe83 in incubator-airflow's branch refs/heads/master
from Dan Fowler
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=ed97204 ]

[AIRFLOW-1104] Update jobs.py so Airflow does not over schedule tasks (#3568)

This change will prevent tasks from getting scheduled and queued over
the concurrency limits set for the dag

> Concurrency check in scheduler should count queued tasks as well as running
> ---------------------------------------------------------------------------
>
>                 Key: AIRFLOW-1104
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1104
>             Project: Apache Airflow
>          Issue Type: Bug
>         Environment: see https://github.com/apache/incubator-airflow/pull/2221
> "Tasks with the QUEUED state should also be counted below, but for now we cannot count
them. This is because there is no guarantee that queued tasks in failed dagruns will or will
not eventually run and queued tasks that will never run will consume slots and can stall a
DAG. Once we can guarantee that all queued tasks in failed dagruns will never run (e.g. make
sure that all running/newly queued TIs have running dagruns), then we can include QUEUED tasks
here, with the constraint that they are in running dagruns."
>            Reporter: Alex Guziel
>            Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message