airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Szakallas <dszakal...@whitepages.com>
Subject Creating dynamic pool from task
Date Thu, 20 Sep 2018 14:07:51 GMT
Hi all,

I have a DAG that creates a cluster, starts computation tasks, and after they completed, tears
down the cluster. I want to limit concurrency for the computation tasks carried on this cluster
to fixed number. So logically, I need a pool that is exclusive to the cluster created by a
task. I don't want interference with other DAGs or different runs of the same DAG.

I thought I could solve this problem by creating a pool dynamically from a task after the
cluster is created and delete it once the computation tasks are finished. I thought I could
template the pool parameter of the computation tasks to make them use this dynamically created
cluster.

However this way the computation tasks will never be triggered. So I think the pool parameter
is saved in the task instance before being templated. I would like to hear your thoughts on
how to achieve the desired behavior.

Thanks,

Dávid Szakállas
Software Engineer | Whitepages Data Services




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message