spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gsvic <>
Subject Are map tasks spilling data to disk?
Date Sun, 15 Nov 2015 18:52:29 GMT
According to  this paper
Spak's map tasks writes the results to disk. 

My actual question is, in  BroadcastHashJoin
doExecute() method at line  109 the mapPartitions
method is called. At this step, Spark will schedule a number of tasks for
execution in order to perform the hash join operation. The results of these
tasks will be written to each worker's disk?

View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message