spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From YanTangZhai <>
Subject [GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Date Thu, 21 Aug 2014 06:50:59 GMT
Github user YanTangZhai commented on the pull request:
    Hi @JoshRosen SparkContext1 creates broadcastManager and initializes HttpBroadcast object.
HttpBroadcast creates httpserver and broadcastDir and so on. However SparkContext2 in the
same process won't initialize HttpBroadcast object when creating broadcastManager. Since HttpBroadcast
object is marked initialized and will not be initialized any more. SparkContext1 and SparkContext2
will share the same HttpBroadcast object. When SparkContext1 stops HttpBroadcast, HttpBroadcast
in SparkContext2 actually is stopped. When HttpBroadcast1 cleans up files, some files owned
by SparkContext2 may be removed. Since they are the same one.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message