spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Assigned] (SPARK-7884) Allow Spark shuffle APIs to be more customizable
Date Wed, 27 May 2015 00:57:18 GMT


Apache Spark reassigned SPARK-7884:

    Assignee:     (was: Apache Spark)

> Allow Spark shuffle APIs to be more customizable
> ------------------------------------------------
>                 Key: SPARK-7884
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Matt Massie
> The current Spark shuffle has some hard-coded assumptions about how shuffle managers
will read and write data.
> The FileShuffleBlockResolver.forMapTask method creates disk writers by calling BlockManager.getDiskWriter.
This forces all shuffle managers to store data using the DiskBlockObjectWriter which read/write
data as record-oriented (preventing column-orient record writing).
> The BlockStoreShuffleFetcher.fetch method relies on the ShuffleBlockFetcherIterator that
assumes shuffle data is written using the BlockManager.getDiskWriter method and doesn't allow
for customization.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message