airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1074) Do not count queued tasks in scheduler concurrency check
Date Thu, 08 Jun 2017 15:37:18 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042871#comment-16042871
] 

ASF subversion and git services commented on AIRFLOW-1074:
----------------------------------------------------------

Commit 70024935f24e0ff3d2861c0ccfa69cdd38084b9d in incubator-airflow's branch refs/heads/v1-8-test
from [~saguziel]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7002493 ]

[AIRFLOW-1074] Don't count queued tasks for concurrency limits

There may be orphaned tasks queued but not in a
running dag run that
will not cleared. We should not count these as
they will interfere.

I hate to do this, but I changed my mind on
counting queued tasks.

1. Queued tasks that are actually queued generally
get set to running pretty quickly.
2. Because of the worker-side check, we won't
actually pass concurrency.

I don't think the queued thing is a big deal
because of this, I'm more worried about orphaned
tasks that are in QUEUED state but not in a
running dag_run (so they wont get reset)
interfering with concurrency.

There may be orphaned tasks queued but not in a
running dag run that
will not cleared. We should not count these as
they will interfere.

Closes #2221 from saguziel/aguziel-concurrency-2


> Do not count queued tasks in scheduler concurrency check
> --------------------------------------------------------
>
>                 Key: AIRFLOW-1074
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1074
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Alex Guziel
>             Fix For: 1.8.2
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message