airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] yrqls21 edited a comment on issue #2986: [AIRFLOW-2027] Only trigger sleep in scheduler after all files have parsed
Date Mon, 13 Aug 2018 19:38:26 GMT
yrqls21 edited a comment on issue #2986: [AIRFLOW-2027] Only trigger sleep in scheduler after
all files have parsed
URL: https://github.com/apache/incubator-airflow/pull/2986#issuecomment-412637527
 
 
   @aoen 
   Agree on sleep for a minimal time interval(max(0, 1s-duration)), it should not be a huge
problem to prolong the scheduler loop time to at least one second every loop( have that in
my branch for a while and looks good until now). 
   
   I'm not so sure about monotonically increasing, what I can see is that if we have parse
files faster than we set file path, then if file was removed/have import error and not updating
the last_finish_time in the 2nd parsing round, we'll still use the last_finish_time from the
1st round and get some non-desirable duration value. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message