airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerard Toonstra (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1133) More tasks than the concurrency limit can run
Date Thu, 27 Apr 2017 14:56:04 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986765#comment-15986765
] 

Gerard Toonstra commented on AIRFLOW-1133:
------------------------------------------

Could be related to this issue too.

> More tasks than the concurrency limit can run
> ---------------------------------------------
>
>                 Key: AIRFLOW-1133
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1133
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Alex Guziel
>
> There are two ways to put checks on dag concurrency:
> 1) The scheduler not queueing too many tasks (via checking the amount of tasks running)
> 2) The worker checking that not too many tasks are running (via the db)
> Right now, both have issues.
> 1 doesn't considered queued tasks which may not be running now, but will be running soon.
Hopefully, check 2 should catch it, but it does not check the condition properly as it only
locks the row, and it seems locking the dag would also be expensive.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message