hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1126) shuffle should use serialization to get comparator
Date Tue, 08 Dec 2009 23:11:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787799#action_12787799

Aaron Kimball commented on MAPREDUCE-1126:

To allow Avro-based comparators during the shuffle, a few changes need to take place. Here's
my survey of the scope of the problem and proposal for addressing it.

HADOOP-6323 allows {{SerializationBase}} to provide {{RawComparator}} instances. But for Avro
to be germane here, the {{Serializer}} and {{Deserializer}} instances used by {{MapOutputBuffer}}
and other shuffle code need to be updated as well. {{MapOutputBuffer}} and other places where
the {{Serializer}} and {{Deserializer}} interfaces are used should be changed to use instead
{{SerializerBase}} and {{DeserializerBase}}. This should be done by changing the current calls
to {{SerializationBase.getSerializer(keyClassName)}} to use the newer {{getSerializer(metadata)}}

Currently the only metadata we store is the {{Serialization-Class}} that identifies the intermediate
key and value classes. This is implicitly set in, e.g. {{Job.setMapOutputKeyClass()}}. (It
actually just stores a class name, and the {{getSerializer(className)}} method converts that
to the appropriate metadata map for {{getSerializer(metadata)}}.) HADOOP-6165 allows an arbitrary
map of metadata to be passed to the serialization framework. We should take advantange of
this map throughout the shuffling process. The job's {{Configuration}} should be updated to
embed a map a la the outline in HADOOP-6165 (See Tom's comment, part 4(ii)) to include a "family"
of configuration parameters. Given the renaming of various parameters, I propose to embed
the metadata maps in {{mapreduce.map.output.key.metadata}} and {{mapreduce.map.output.value.metadata}}.

The {{JobContext.setMapOutputKeyClass()}} method would instead of setting {{mapreduce.map.output.key.class}},
set the appropriate metadata strings under {{mapreduce.map.output.key.metadata}} or {{mapreduce.map.output.value.metadata}}
to use the {{WritableSerialization}} implementation. If this metadata is not present, metadata
could be inferred from {{mapreduce.map.output.key/value.class}}, which should be marked as

{{JobContext}} would also include new methods which could be used instead of {{setMapOutputKeyClass()}},
e.g. {{setMapOutputKeySchema()}} which would accept an Avro schema and configure the shuffle
to use AvroGenericSerialization for this data. A generic {{setMapOutputKeySerializationMetadata()}}
could be added as well, which would accept any {{string -> string}} map of metadata for
use with other arbitrary {{SerializationBase}} implementations. (It might be used internally
by {{setMapOutputKeyClass()}} and {{setMapOutputKeySchema()}}.)

On the internal side, {{JobContext}} already provides a method {{getSortComparator()}} which
provides the appropriate {{RawComparator}} implementation. But it does not directly provide
{{SerializerBase}} or {{DeserializerBase}} instances; it instead only returns the key and
value class names and clients of this (e.g., {{MapOutputBuffer}}) are expected to instantiate
the correct serialization objects themselves. I think that {{JobContext}} should provide {{getMapOutputKeySerializer()}}
and {{getMapOutputKeyDeserializer()}} methods that correctly handle either the use of {{mapreduce.map.output.key.metadata}}
or {{mapreduce.map.output.key.class}} to avoid duplication of this logic.

All {{SerializerBase}}, {{DeserializerBase}}, and {{RawComparator}} implementations instantiated
during shuffle would then be created via uses of this new API.

Longer term, metadata maps are necessary for reducer output as well; e.g., {{mapreduce.reduce.output.key.metadata}}.
This would allow full end-to-end Avro use for user data. Similar methods for JobContext would
need to be added to set these parameters, and {{Job.setOutputKeyClass()}} et al. would need
to be updated to use this new API as well. But I think that this may be more reasonably part
of MAPREDUCE-815 than the current issue, which is concerned with just the shuffle process.

> shuffle should use serialization to get comparator
> --------------------------------------------------
>                 Key: MAPREDUCE-1126
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task
>            Reporter: Doug Cutting
> Currently the key comparator is defined as a Java class.  Instead we should use the Serialization
API to create key comparators.  This would permit, e.g., Avro-based comparators to be used,
permitting efficient sorting of complex data types without having to write a RawComparator
in Java.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message