spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] advancedxy commented on a change in pull request #26085: [SPARK-29434] [Core] Improve the MapStatuses Serialization Performance
Date Fri, 11 Oct 2019 03:06:48 GMT
advancedxy commented on a change in pull request #26085: [SPARK-29434] [Core] Improve the MapStatuses
Serialization Performance
URL: https://github.com/apache/spark/pull/26085#discussion_r333810252
 
 

 ##########
 File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala
 ##########
 @@ -807,13 +808,17 @@ private[spark] object MapOutputTracker extends Logging {
   private val BROADCAST = 1
 
   // Serialize an array of map output locations into an efficient byte format so that we
can send
-  // it to reduce tasks. We do this by compressing the serialized bytes using GZIP. They
will
+  // it to reduce tasks. We do this by compressing the serialized bytes using Zstd. They
will
   // generally be pretty compressible because many map outputs will be on the same hostname.
   def serializeMapStatuses(statuses: Array[MapStatus], broadcastManager: BroadcastManager,
       isLocal: Boolean, minBroadcastSize: Int): (Array[Byte], Broadcast[Array[Byte]]) = {
-    val out = new ByteArrayOutputStream
-    out.write(DIRECT)
-    val objOut = new ObjectOutputStream(new GZIPOutputStream(out))
+    import scala.language.reflectiveCalls
+    val out = new ByteArrayOutputStream(4096) {
 
 Review comment:
   Just curious, is there any specific reason to choose 4096 here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message