airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramandumcs@gmail.com <ramandu...@gmail.com>
Subject Dynamic pool Allocation policy for a Task
Date Wed, 29 May 2019 06:15:30 GMT
Hi All,

Currently DAG tasks are assigned to a pool during DAG creation time so its kind of static
pool assignment which is done at creation time and might lead to unused resources if there
are multiple pools configured in the airflow.
So does it make sense to have support for pool allocation policy where a task can be assigned
to some other pool if current pre-allocated pool resources are not available. There can be
some limit on how much such slots can be used.
We have a use case where we have multiple pools which can better be utilised with some reallocation.

Thanks,
Raman Gupta
 
 


Mime
View raw message