spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <>
Subject [jira] [Created] (SPARK-9808) Remove hash shuffle file consolidation
Date Tue, 11 Aug 2015 01:32:46 GMT
Josh Rosen created SPARK-9808:

             Summary: Remove hash shuffle file consolidation
                 Key: SPARK-9808
             Project: Spark
          Issue Type: Improvement
          Components: Shuffle, Spark Core
            Reporter: Josh Rosen
            Assignee: Josh Rosen

I think that we should remove {{spark.shuffle.consolidateFiles}} and its associated implementation
for Spark 1.5.0. This feature isn't properly tested and does not work with the external shuffle
service. Since it's likely to be buggy and since its motivation has been subsumed by sort-based
shuffle, I think it's a prime candidate for removal now.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message