airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Capwell <dcapw...@gmail.com>
Subject Pools and extra capacity?
Date Thu, 17 Aug 2017 05:47:02 GMT
I'm looking into pools and had a few questions

Let's say I have two pools, each of 50% of the cluster.  If one pool is at
capacity and has a backlog, but the other pool is idle, will airflow allow
the first pool's work to start consuming the slots from the idle pool, and
if so is their preemption to rectify if needed? Since the docs don't cover
this I expect that this is not supported?

Also, let's say I have a DAG that I need to always run regardless of if
there are free slot or not; how could I accomplish this? Best I can see is
pools and make one just slightly smaller than the full cluster.

Thanks for your time reading this email

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message