airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerard Toonstra (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-1133) More tasks than the concurrency limit can run
Date Thu, 27 Apr 2017 14:56:04 GMT


Gerard Toonstra commented on AIRFLOW-1133:

Could be related to this issue too.

> More tasks than the concurrency limit can run
> ---------------------------------------------
>                 Key: AIRFLOW-1133
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Alex Guziel
> There are two ways to put checks on dag concurrency:
> 1) The scheduler not queueing too many tasks (via checking the amount of tasks running)
> 2) The worker checking that not too many tasks are running (via the db)
> Right now, both have issues.
> 1 doesn't considered queued tasks which may not be running now, but will be running soon.
Hopefully, check 2 should catch it, but it does not check the condition properly as it only
locks the row, and it seems locking the dag would also be expensive.

This message was sent by Atlassian JIRA

View raw message