From dev-return-6198-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Tue Aug 21 17:40:26 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id F1478180630 for ; Tue, 21 Aug 2018 17:40:25 +0200 (CEST) Received: (qmail 95872 invoked by uid 500); 21 Aug 2018 15:40:20 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 95388 invoked by uid 99); 21 Aug 2018 15:40:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Aug 2018 15:40:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 27990C96DC for ; Tue, 21 Aug 2018 15:40:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.869 X-Spam-Level: ** X-Spam-Status: No, score=2.869 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id sGdDQIUoUkrn for ; Tue, 21 Aug 2018 15:40:18 +0000 (UTC) Received: from mail-oi0-f50.google.com (mail-oi0-f50.google.com [209.85.218.50]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5B1AE5F416 for ; Tue, 21 Aug 2018 15:40:17 +0000 (UTC) Received: by mail-oi0-f50.google.com with SMTP id v8-v6so32791451oie.5 for ; Tue, 21 Aug 2018 08:40:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=xU4yow9bFxWbh3vIBz7Rsj/rJ/f8VapaOwwS1y1iSx4=; b=S+7XQs/hvY76s8cSOtQLkTMSECH4nXwikq6euPIxshNNCC4DkadsQfLTPE6woUSt3t KTOSk1j9wnqJbjQ/3dARxzBhvbFRTfbh9yZJGEW9SaDk+45vuwvVaKM5aF4BgDNxRBKU /zY8E2VtE9T3HaCvP8o530m8mA9b5BK9zFLZurx5i+trJzfRRDCHN1+Cs06NfovBgEdP +JlgpefHwJ57qjGVxCUZ8gvB4OmXwakYS8hNcWfPj0c9bxilzsCEMhLo/DxtNZ1J0MUe HvNumLLJoxLavWAbml6vqm4Ud3i6tnfeyqH2XhYVcIgUqu8hm62FiNS9604aWLF0Bj4f wq7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xU4yow9bFxWbh3vIBz7Rsj/rJ/f8VapaOwwS1y1iSx4=; b=XTPbwf7GNbUd17rjltQTnwuKAO6X32IOEQ//qTtwLVOxmY7DxyhSPvt4wI03BLQ+0h elVwnxVDUYDF1nubc2xC7oDxBwcwdo/w5Q43YSb6prSC8AwlglyksgW6q6fVEWzV62ss dZicCJR3ZessawykYMNw8nYEMp1pT6C79SnlfEeQCOUkT/HOVslWHUtz5Os1B8dQyFjt PkSMwYrcVYZBPBcL8O07MxyPhG4JQ8DAYHH5SULZ/lXUBn6iLQc147xNDC7h0y1G/jVu 2nuWjk3lLNp5H/+Y0ee96GrF0UoYsyyE/GN9598LkJjHkkKo9eZXIGtaNLdhduz7KsnR tihA== X-Gm-Message-State: APzg51B0zt66mfcjEC3BYXTtwZdo6vz5vHfltR0K5yPQLxcouDhbvKew KoNAi9XSgrvlLo8bRBKFGIkE7HB/c13+WO0KnzI5BbQ8 X-Google-Smtp-Source: AA+uWPyoLX8do8gTwvrmHLh5f4mDzEJM3etgDavvjbgmHVAQURYWgTbHtWquhr5YftO3P60TbYOCk0uZlTxqoQWfXQ0= X-Received: by 2002:aca:6b15:: with SMTP id g21-v6mr11498423oic.149.1534866015806; Tue, 21 Aug 2018 08:40:15 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: raman gupta Date: Tue, 21 Aug 2018 21:11:14 +0530 Message-ID: Subject: Re: Task is Stuck in Up_For_Retry To: dev@airflow.incubator.apache.org Cc: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="000000000000a66dff0573f3d7a4" --000000000000a66dff0573f3d7a4 Content-Type: text/plain; charset="UTF-8" One possibility is the unavailability of session while calling self.task_instance._check_and_change_state_before_execution function. (Session is provided via @provide_session decorator) On Tue, Aug 21, 2018 at 7:09 PM vardanguptacse@gmail.com < vardanguptacse@gmail.com> wrote: > Is there any possibility that on call of function > _check_and_change_state_before_execution at > https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/jobs.py#L2500, > this method is not actually being called > https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/models.py#L1299? > Because even in a happy scenario, no logs is printed from method's > implementation and directly control is reaching here > https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/jobs.py#L2512 > while in stuck phase, we are seeing this log > https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/jobs.py#L2508 > i.e. Task is not able to be run, FYI we've not set any sort of dependency > with dag. > > Regards, > Vardan Gupta > > On 2018/08/16 08:25:37, ramandumcs@gmail.com > wrote: > > Hi All, > > > > We are using airflow 1.9 with Local Executor more. Intermittently we are > observing that tasks are getting stuck in "up_for_retry" mode and are > getting retried again and again exceeding their configured max retries > count. like we have configured max retries as 2 but task is retried 15 > times and got stuck in up_for_retry state. > > Any pointer on this would be helpful. > > > > Thanks, > > Raman Gupta > > > --000000000000a66dff0573f3d7a4--