spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject MapOutputTracker.getMapSizesByExecutorId and mutation on the driver?
Date Fri, 23 Dec 2016 14:13:35 GMT
Hi,

I've been reviewing how MapOutputTracker works and can't understand
the comment [1]:

// Synchronize on the returned array because, on the driver, it gets
mutated in place

How is this possible since "the returned array" is a local value? I'm
stuck and would appreciate help. Thanks!

(It also says "Called from executors" [2] so how could the driver be involved?!)

[1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L145

[2] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L133

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message