airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
Date Fri, 12 Oct 2018 08:06:39 GMT
KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler
loop
URL: https://github.com/apache/incubator-airflow/pull/3873#issuecomment-429241610
 
 
   @Fokko Thanks for reviewing. I actually created [a ticket](https://issues.apache.org/jira/browse/AIRFLOW-3194)
for further refactoring in the code base to use with block as there're a lot such cases and
I feel it is beyond the scope of this PR to change all of them( updated all use cases that
I touched).
   
   Would you actually elaborate a bit more on the "weird stuff in the number of connections"
please? Might be something we didn't catch.
   
   Also this reminds me about one thing: with the new logic, it is more likely that the scheduler
will leave orphan processes in extreme cases, e.g. for some reason the agent process crashed(or
got a SIGTERM) or cannot gracefully exit, the dag parsing processes will be left behind. Should
we put a note somewhere about this? Actually we already have similar problem, e.g. scheduler
process got SIGTERM. Do we just expect people to know this and clean up the left over processes?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message