spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-7413) Time to write shuffle spill files is not captured in ShuffleWriteMetrics
Date Wed, 06 May 2015 23:44:59 GMT
Josh Rosen created SPARK-7413:
---------------------------------

             Summary: Time to write shuffle spill files is not captured in ShuffleWriteMetrics
                 Key: SPARK-7413
                 URL: https://issues.apache.org/jira/browse/SPARK-7413
             Project: Spark
          Issue Type: Bug
          Components: Shuffle
            Reporter: Josh Rosen


In ExternalSorter's {{spillToMergeableFile()}} method, we pass ShuffleWriteMetrics instances
to the disk writers, but discard the {{shuffleWriteTime}} metrics captured here.  I think
that we should account for this IO time, possibly by introducing new metrics to distinguish
time spent writing spills vs. writing final shuffle output and extending the UI to break down
the overall IO write time in terms of these two components.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message