airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2895) Prevent scheduler from spamming heartbeats/logs
Date Mon, 13 Aug 2018 18:03:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578723#comment-16578723
] 

ASF GitHub Bot commented on AIRFLOW-2895:
-----------------------------------------

aoen opened a new pull request #3747: [AIRFLOW-2895] Prevent scheduler from spamming heartbeats/logs
URL: https://github.com/apache/incubator-airflow/pull/3747
 
 
   Reverts most of AIRFLOW-2027 until the issues with it can be fixed.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
     - https://issues.apache.org/jira/browse/AIRFLOW-2895
   
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI changes:
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely
good reason:
   Reverting some broken code that is missing tests.
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I have squashed
multiple commits if they address the same issue. In addition, my commits follow the guidelines
from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X] In case of new functionality, my PR adds documentation that describes how to use
it.
     - When adding new operators/hooks/sensors, the autoclass documentation generation needs
to be added.
   
   ### Code Quality
   
   - [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   
   @bolkedebruin 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Prevent scheduler from spamming heartbeats/logs
> -----------------------------------------------
>
>                 Key: AIRFLOW-2895
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2895
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: Dan Davydov
>            Assignee: Dan Davydov
>            Priority: Major
>
> There seems to be a couple of problems with [https://github.com/apache/incubator-airflow/pull/2986]
that cause the sleep to not trigger and Scheduler heartbeating/logs to be spammed:
>  # If all of the files are being processed in the queue, there is no sleep (can be fixed
by sleeping for min_sleep even if there are no files)
>  # I have heard reports that some files can return a parsing time that is monotonically
increasing for some reason (e.g. file actually parses in 1s each loop, but the reported duration
seems to use the very time the file was parsed as the start time instead of the last time),
I haven't confirmed this but it sounds problematic.
> To unblock the release I'm reverting this PR for now. It should be re-added with tests/mocking.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message