airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rhodes (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-41) SubdagOperators can oversubscribe to pools due to race condition
Date Thu, 05 May 2016 10:18:12 GMT


Paul Rhodes commented on AIRFLOW-41:

+1 on the PoolHandler suggestion.

I've had discussions with [~jlowin] in the past about extending how pools work mainly in order
to support dynamic resource allocation/deallocation associated with such a pool (think AWS
resources as a good example).

1. An operator knows what slot it has (rather than just a lookup on a pool_id field in the
2. Pool hooks could then be defined on pool slot allocation/deallocation and maybe a heartbeat
to tidyup unused slots.

> SubdagOperators can oversubscribe to pools due to race condition
> ----------------------------------------------------------------
>                 Key: AIRFLOW-41
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler, subdag
>    Affects Versions: Airflow 1.7.1
>            Reporter: Bolke de Bruin
> SubDagOperators essentially create their own mini-scheduler. Which can interfere with
the main scheduler. 
> SubdagOperators check if there is slot available in a Pool. However this slot is not
claimed at the same time leaving room for main scheduler to also check for the slot. Both
can then obtain a slot and thus oversubscribe
> A solution could be a centralized PoolHandler that gives out slots

This message was sent by Atlassian JIRA

View raw message