From commits-return-81078-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Fri Dec 13 09:50:06 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 48B8218067C for ; Fri, 13 Dec 2019 10:50:06 +0100 (CET) Received: (qmail 28902 invoked by uid 500); 13 Dec 2019 09:50:05 -0000 Mailing-List: contact commits-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list commits@airflow.apache.org Received: (qmail 28884 invoked by uid 99); 13 Dec 2019 09:50:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Dec 2019 09:50:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E665CC0809 for ; Fri, 13 Dec 2019 09:50:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -112.989 X-Spam-Level: X-Spam-Status: No, score=-112.989 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_DMARC_STATUS=0.01, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id OkOE6lSWSefF for ; Fri, 13 Dec 2019 09:50:02 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=207.244.88.153; helo=mail.apache.org; envelope-from=jira@apache.org; receiver= Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with SMTP id 548BCBC545 for ; Fri, 13 Dec 2019 09:50:02 +0000 (UTC) Received: (qmail 28843 invoked by uid 99); 13 Dec 2019 09:50:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Dec 2019 09:50:02 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id E298DE2F61 for ; Fri, 13 Dec 2019 09:50:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 2E96278027F for ; Fri, 13 Dec 2019 09:50:00 +0000 (UTC) Date: Fri, 13 Dec 2019 09:50:00 +0000 (UTC) From: "Scato Eggen (Jira)" To: commits@airflow.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AIRFLOW-6252) DAG.clear() does not clear certain task instances MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AIRFLOW-6252?page=3Dcom.atlass= ian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scato Eggen updated AIRFLOW-6252: --------------------------------- Description:=20 Use case: When running a task from a unit test, I need to clear the DAG as demonstrat= ed in=C2=A0[https://blog.godatadriven.com/testing-and-debugging-apache-airf= low]. Observed behavior: At first, everything worked as expected, but at some point the task wouldn'= t start anymore. Upon inspection, it turned out that: * the task instance had status RUNNING but no job_id * airflow.models.taskinstance.clear_task_instances skips tasks without a j= ob_id, so it was keeping the task instance in state RUNNING * as a result TaskInstance._check_and_change_state_before_execution starte= d returning False A factor that could have contributed was that changed the task from having = retries=3D1 to retries=3D0. Also, I was stepping through the code with the = debugger. Expected behavior: DAG.clear() clears all task instances, even the ones that have weird states= . Workaround: Delete ~/airflow/airflow.db and rerun airflow initdb. =C2=A0 was: Use case: When running a task from a unit test, I need to clear the DAG as demonstrat= ed in=C2=A0[https://blog.godatadriven.com/testing-and-debugging-apache-airf= low]. Observed behavior: At first, everything worked as expected, but at some point the task wouldn'= t start anymore. Upon inspection, it turned out that the task instance had = status RUNNING but no job_id. airflow.models.taskinstance.clear_task_instan= ces skips tasks without a job_id, so it was keeping the task instance in st= ate RUNNING and as a result TaskInstance._check_and_change_state_before_exe= cution started returning False. A factor that could have contributed was that changed the task from having = retries=3D1 to retries=3D0. Also, I was stepping through the code with the = debugger. Expected behavior: DAG.clear() clears all task instances, even the ones that have weird states= . Workaround: Delete ~/airflow/airflow.db and rerun airflow initdb. =C2=A0 > DAG.clear() does not clear certain task instances > ------------------------------------------------- > > Key: AIRFLOW-6252 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6252 > Project: Apache Airflow > Issue Type: Bug > Components: DAG > Affects Versions: 1.10.3 > Reporter: Scato Eggen > Priority: Major > > Use case: > When running a task from a unit test, I need to clear the DAG as demonstr= ated in=C2=A0[https://blog.godatadriven.com/testing-and-debugging-apache-ai= rflow]. > Observed behavior: > At first, everything worked as expected, but at some point the task would= n't start anymore. Upon inspection, it turned out that: > * the task instance had status RUNNING but no job_id > * airflow.models.taskinstance.clear_task_instances skips tasks without a= job_id, so it was keeping the task instance in state RUNNING > * as a result TaskInstance._check_and_change_state_before_execution star= ted returning False > A factor that could have contributed was that changed the task from havin= g retries=3D1 to retries=3D0. Also, I was stepping through the code with th= e debugger. > Expected behavior: > DAG.clear() clears all task instances, even the ones that have weird stat= es. > Workaround: > Delete ~/airflow/airflow.db and rerun airflow initdb. > =C2=A0 -- This message was sent by Atlassian Jira (v8.3.4#803005)