From dev-return-8570-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Wed May 29 06:15:38 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7590818064C for ; Wed, 29 May 2019 08:15:38 +0200 (CEST) Received: (qmail 15370 invoked by uid 500); 29 May 2019 06:15:36 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 15358 invoked by uid 99); 29 May 2019 06:15:36 -0000 Received: from ui-eu-01.ponee.io (HELO localhost) (176.9.59.70) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 May 2019 06:15:36 +0000 MIME-Version: 1.0 References: Date: Wed, 29 May 2019 06:15:30 -0000 From: ramandumcs@gmail.com X-Mailer: LuaSocket 3.0-rc1 Message-ID: Content-Type: text/plain; charset=utf-8 x-ponymail-agent: PonyMail Composer/0.3 To: In-Reply-To: Subject: Dynamic pool Allocation policy for a Task x-ponymail-sender: d07766641e2bf63b1ee342486ca9eaa753e59524 Hi All, Currently DAG tasks are assigned to a pool during DAG creation time so its kind of static pool assignment which is done at creation time and might lead to unused resources if there are multiple pools configured in the airflow. So does it make sense to have support for pool allocation policy where a task can be assigned to some other pool if current pre-allocated pool resources are not available. There can be some limit on how much such slots can be used. We have a use case where we have multiple pools which can better be utilised with some reallocation. Thanks, Raman Gupta