From dev-return-5669-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Wed Jul 25 18:02:45 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 34EE118062C for ; Wed, 25 Jul 2018 18:02:45 +0200 (CEST) Received: (qmail 38776 invoked by uid 500); 25 Jul 2018 16:02:44 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 38760 invoked by uid 99); 25 Jul 2018 16:02:43 -0000 Received: from ui-eu-01.ponee.io (HELO localhost) (176.9.59.70) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Jul 2018 16:02:43 +0000 Subject: Pause DAG leading to Hang State of DAG Date: Wed, 25 Jul 2018 16:02:41 -0000 x-ponymail-agent: PonyMail Composer/0.3 X-Mailer: LuaSocket 3.0-rc1 In-Reply-To: To: x-ponymail-sender: f0f47dab6a5f100bb8728c720a9b00c0c5680e07 Message-ID: References: Content-Type: text/plain; charset=utf-8 MIME-Version: 1.0 From: vardanguptacse@gmail.com Hi Everyone, We are using Airflow v1.9 and observing a pattern where we have few DAGs marked in paused state and scheduled existing DAG runs are not getting marked success/failure instead they are always in running state. The moment, we unpause the DAG, it will back fill all the missed executions along with the changing state of existing DAG runs in dag_run table. Since, we've written a DELETE functionality in an abstracted form which works only if the DAG is disabled & no active runs are there, we could not perform DELETE operation as runs are in running state forever in such scenarios. Is it a default behavior that if a DAG is disabled then no further dag run will be kicked off along with no execution of residual part of running instances of DAG(may be 2/3 task instances) which was unprocessed by the time paused state was requested? If this is a bug, we can open JIRA and also contribute. Let us know, if additional information is required. Would love to get this addressed.