airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rhodes (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-41) SubdagOperators can oversubscribe to pools due to race condition
Date Thu, 05 May 2016 10:18:12 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272161#comment-15272161
] 

Paul Rhodes commented on AIRFLOW-41:
------------------------------------

+1 on the PoolHandler suggestion.

I've had discussions with [~jlowin] in the past about extending how pools work mainly in order
to support dynamic resource allocation/deallocation associated with such a pool (think AWS
resources as a good example).

1. An operator knows what slot it has (rather than just a lookup on a pool_id field in the
scheduler)
2. Pool hooks could then be defined on pool slot allocation/deallocation and maybe a heartbeat
to tidyup unused slots.


> SubdagOperators can oversubscribe to pools due to race condition
> ----------------------------------------------------------------
>
>                 Key: AIRFLOW-41
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-41
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler, subdag
>    Affects Versions: Airflow 1.7.1
>            Reporter: Bolke de Bruin
>
> SubDagOperators essentially create their own mini-scheduler. Which can interfere with
the main scheduler. 
> SubdagOperators check if there is slot available in a Pool. However this slot is not
claimed at the same time leaving room for main scheduler to also check for the slot. Both
can then obtain a slot and thus oversubscribe
> A solution could be a centralized PoolHandler that gives out slots



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message