From commits-return-20503-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Sun Sep 2 20:08:14 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 18D42180651 for ; Sun, 2 Sep 2018 20:08:13 +0200 (CEST) Received: (qmail 69232 invoked by uid 500); 2 Sep 2018 18:08:07 -0000 Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list commits@airflow.incubator.apache.org Received: (qmail 69159 invoked by uid 99); 2 Sep 2018 18:08:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Sep 2018 18:08:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 78C77C1D3D for ; Sun, 2 Sep 2018 18:08:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 5VR1BzAfR30R for ; Sun, 2 Sep 2018 18:08:06 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 6CB5D5F5AB for ; Sun, 2 Sep 2018 18:08:05 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id D36FDE2668 for ; Sun, 2 Sep 2018 18:08:03 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 11D5A26B96 for ; Sun, 2 Sep 2018 18:08:03 +0000 (UTC) Date: Sun, 2 Sep 2018 18:08:03 +0000 (UTC) From: "Apache Spark (JIRA)" To: commits@airflow.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Assigned] (AIRFLOW-2895) Prevent scheduler from spamming heartbeats/logs MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AIRFLOW-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned AIRFLOW-2895: ------------------------------------- Assignee: Dan Davydov (was: Holden Karau's magical unicorn) > Prevent scheduler from spamming heartbeats/logs > ----------------------------------------------- > > Key: AIRFLOW-2895 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2895 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler > Reporter: Dan Davydov > Assignee: Dan Davydov > Priority: Major > > There seems to be a couple of problems with [https://github.com/apache/incubator-airflow/pull/2986] that cause the sleep to not trigger and Scheduler heartbeating/logs to be spammed: > # If all of the files are being processed in the queue, there is no sleep (can be fixed by sleeping for min_sleep even if there are no files) > # I have heard reports that some files can return a parsing time that is monotonically increasing for some reason (e.g. file actually parses in 1s each loop, but the reported duration seems to use the very time the file was parsed as the start time instead of the last time), I haven't confirmed this but it sounds problematic. > To unblock the release I'm reverting this PR for now. It should be re-added with tests/mocking. -- This message was sent by Atlassian JIRA (v7.6.3#76005)