spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] manuzhang commented on issue #27998: [SPARK-31219][YARN] Enable closeIdleConnections in YarnShuffleService
Date Thu, 26 Mar 2020 01:53:15 GMT
manuzhang commented on issue #27998: [SPARK-31219][YARN] Enable closeIdleConnections in YarnShuffleService
URL: https://github.com/apache/spark/pull/27998#issuecomment-604182314
 
 
   @xuanyuanking @tgravescs
   Here are detailed timelines of our investigation.
   
   1. We found connections on our clusters building up continuously (> 10k for some nodes).
Is that normal ? We don't think so.
   2. We looked into the connections on one node and found there were a lot of half-open connections.
(connections only existed on one node)
   3. We also checked those connections were very old (> 21 hours). (FYI, https://superuser.com/questions/565991/how-to-determine-the-socket-connection-up-time-on-linux)
   4. Looking at the code,  `TransportContext` registers an `IdleStateHandler` which should
fire an `IdleStateEvent` when timeout. We did a heap dump of the `YarnShuffleService` and
checked the attributes of `IdleStateHandler`. It turned out `firstAllIdleEvent` of many `IdleStateHandler`s
were already `false` so `IdleStateEvent` were already fired.
   5. Finally, we realized the `IdleStateEvent` would not be handled since `closeIdleConnections`
are hardcoded to `false` for `YarnShuffleService`.
   
   The above is based on what we've seen and know. Please correct me if any understanding
is wrong or inaccurate.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message