airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dennis O'Brien" <den...@dennisobrien.net>
Subject Re: Assigning multiple pools to a task
Date Sat, 12 May 2018 21:28:25 GMT
A task is assigned to a pool by the task specifying the name of the pool.
The docs suggest that the pool argument is a string, not a list of strings.
https://airflow.apache.org/code.html#baseoperator

And looking at the code it does seem like this relationship of one task
assigned to zero or one pools is strict.

It's an interesting idea of trying to associated a task with multiple
pools.  I think things would get complicated fast to avoid deadlock
situations.  Do you know of any other work flow systems that support
one-to-many relationship?

Is it possible to bundle the resources into a single resource?  I'm
guessing this is not an option, and that tasks use different combinations
of these resources.

cheers,
Dennis


On Fri, May 11, 2018 at 1:04 PM Neelesh <neeleshs@gmail.com> wrote:

> Hello,
>
> Is it possible to assign multiple pools to a task?
>
> I'm wrapping a bash operator around some heavy-weight custom data
> processing script that cannot be modified. This script cannot be modified,
> and it uses multiple resources. Im trying to use pools as proxies for these
> resources.
>
> Thanks
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message