airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: Pools and extra capacity?
Date Thu, 17 Aug 2017 19:36:40 GMT
The point of pools is to limit parallelism on a logical set of tasks
instances to a certain number. Overflowing into another pool would break
the only guarantee it provides.

`priority_weigth` works along with pool to define which task should be
scheduled first once slots open up. It won't kill any other tasks if higher
priority tasks show up, it just re-orders the queue.

Max

On Wed, Aug 16, 2017 at 10:47 PM, David Capwell <dcapwell@gmail.com> wrote:

> I'm looking into pools and had a few questions
>
> Let's say I have two pools, each of 50% of the cluster.  If one pool is at
> capacity and has a backlog, but the other pool is idle, will airflow allow
> the first pool's work to start consuming the slots from the idle pool, and
> if so is their preemption to rectify if needed? Since the docs don't cover
> this I expect that this is not supported?
>
> Also, let's say I have a DAG that I need to always run regardless of if
> there are free slot or not; how could I accomplish this? Best I can see is
> pools and make one just slightly smaller than the full cluster.
>
> Thanks for your time reading this email
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message