spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle
Date Fri, 10 Apr 2015 18:21:13 GMT

     [ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-2926:
-----------------------------------

    Assignee: Apache Spark  (was: Saisai Shao)

> Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle
> ------------------------------------------------------------------
>
>                 Key: SPARK-2926
>                 URL: https://issues.apache.org/jira/browse/SPARK-2926
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle
>    Affects Versions: 1.1.0
>            Reporter: Saisai Shao
>            Assignee: Apache Spark
>         Attachments: SortBasedShuffleRead.pdf, Spark Shuffle Test Report(contd).pdf,
Spark Shuffle Test Report.pdf
>
>
> Currently Spark has already integrated sort-based shuffle write, which greatly improve
the IO performance and reduce the memory consumption when reducer number is very large. But
for the reducer side, it still adopts the implementation of hash-based shuffle reader, which
neglects the ordering attributes of map output data in some situations.
> Here we propose a MR style sort-merge like shuffle reader for sort-based shuffle to better
improve the performance of sort-based shuffle.
> Working in progress code and performance test report will be posted later when some unit
test bugs are fixed.
> Any comments would be greatly appreciated. 
> Thanks a lot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message