airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Szakallas <>
Subject Creating dynamic pool from task
Date Thu, 20 Sep 2018 14:07:51 GMT
Hi all,

I have a DAG that creates a cluster, starts computation tasks, and after they completed, tears
down the cluster. I want to limit concurrency for the computation tasks carried on this cluster
to fixed number. So logically, I need a pool that is exclusive to the cluster created by a
task. I don't want interference with other DAGs or different runs of the same DAG.

I thought I could solve this problem by creating a pool dynamically from a task after the
cluster is created and delete it once the computation tasks are finished. I thought I could
template the pool parameter of the computation tasks to make them use this dynamically created

However this way the computation tasks will never be triggered. So I think the pool parameter
is saved in the task instance before being templated. I would like to hear your thoughts on
how to achieve the desired behavior.


Dávid Szakállas
Software Engineer | Whitepages Data Services

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message