Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAB=riaBKB9HbF93iOO3tDdWORN1Sm8kPCQA7fgBqhmD6ZeQiEA@mail.gmail.com>
References: <CAB=riaA_wteYAytjy+yEPTxpqKf7on2fXtQkjhou0+Rjcbnh=w@mail.gmail.com>
 <CAB=riaD83-Ot35RLBOu2kGKPhCtD8meSWebaAcN79ZvWaRui4w@mail.gmail.com>
 <CAB=riaADeSuO_kra7GQdOSvjygX0E=Bon6i6meRB3TW+KuZDbA@mail.gmail.com>
 <CAB=riaBtvsLbCgDKNArhASvV5Vn3PWPEBTTOFfQO2r+r3nLbGA@mail.gmail.com>
 <CAB=riaDOhTBdOnLg11w7=hM29qS5KFDMrvwy=rmDVi-dYmiVEg@mail.gmail.com>
 <CAB=riaBaCzNimxKJ0zVDe24THZ==QQe5V3m9v2mRBJ=evuX6mA@mail.gmail.com>
 <CAB=riaAfqwzObK5T8BoBQ95DpHh8S=C7xJoYj_U8BQA_bmSK3A@mail.gmail.com>
 <CAB=riaDaLkDTtzjdn8fbM1nha8Ls-01sNS62c9h=oaO=jqRggg@mail.gmail.com>
 <CAB=riaDEg+P_ds_RGzWetPPG=shaDs=hrXyY=Y4RCRq-jpqn1A@mail.gmail.com>
 <CAB=riaB9vZ4DJkpFcuT897CQjOhV73wJpoURn-xLsysCJ1QYCw@mail.gmail.com>
 <CAB=riaAarxwBhFpehqEaP9w3iHC+qM9jSoC8h8N45TnUeuwTtg@mail.gmail.com>
 <CAB=riaCUDcfKm9PS8cB_He8Beb+BWAvnNTme5Jn_bvUqDU3Ogw@mail.gmail.com> <CAB=riaBKB9HbF93iOO3tDdWORN1Sm8kPCQA7fgBqhmD6ZeQiEA@mail.gmail.com>
From: Maxime Beauchemin <maximebeauchemin@gmail.com>
Date: Thu, 17 Aug 2017 12:36:40 -0700
Message-ID: <CAHEEp7WbSkStoEoPE5SZ_0F7B96FmZXM36y=e=h+_rA46LB_Tw@mail.gmail.com>
Subject: Re: Pools and extra capacity?
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary="94eb2c13be6eb5c0fd0556f82126"
archived-at: Thu, 17 Aug 2017 19:36:46 -0000

--94eb2c13be6eb5c0fd0556f82126
Content-Type: text/plain; charset="UTF-8"

The point of pools is to limit parallelism on a logical set of tasks
instances to a certain number. Overflowing into another pool would break
the only guarantee it provides.

`priority_weigth` works along with pool to define which task should be
scheduled first once slots open up. It won't kill any other tasks if higher
priority tasks show up, it just re-orders the queue.

Max

On Wed, Aug 16, 2017 at 10:47 PM, David Capwell <dcapwell@gmail.com> wrote:

> I'm looking into pools and had a few questions
>
> Let's say I have two pools, each of 50% of the cluster.  If one pool is at
> capacity and has a backlog, but the other pool is idle, will airflow allow
> the first pool's work to start consuming the slots from the idle pool, and
> if so is their preemption to rectify if needed? Since the docs don't cover
> this I expect that this is not supported?
>
> Also, let's say I have a DAG that I need to always run regardless of if
> there are free slot or not; how could I accomplish this? Best I can see is
> pools and make one just slightly smaller than the full cluster.
>
> Thanks for your time reading this email
>

--94eb2c13be6eb5c0fd0556f82126--