spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Assigned] (SPARK-9043) Serialize key, value and combiner classes in ShuffleDependency
Date Tue, 14 Jul 2015 21:15:05 GMT


Apache Spark reassigned SPARK-9043:

    Assignee: Apache Spark

> Serialize key, value and combiner classes in ShuffleDependency
> --------------------------------------------------------------
>                 Key: SPARK-9043
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Matt Massie
>            Assignee: Apache Spark
> ShuffleManager implementations are currently not given type information regarding the
key, value and combiner classes. Serialization of shuffle objects relies on them being JavaSerializable,
with methods defined for reading/writing the object or, alternatively, serialization via Kryo
which uses reflection.
> Serialization systems like Avro, Thrift and Protobuf generate classes with zero argument
constructors and explicit schema information (e.g. IndexedRecords in Avro have get, put and
getSchema methods).
> By serializing the key, value and combiner class names in ShuffleDependency, shuffle
implementations will have access to schema information when registerShuffle() is called.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message