Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <1717716289.1260313880133.JavaMail.jira@brutus>
Date: Tue, 8 Dec 2009 23:11:20 +0000 (UTC)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Commented: (MAPREDUCE-1126) shuffle should use serialization
 to get comparator
In-Reply-To: <367401715.1256058959495.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787799#action_12787799 ] 

Aaron Kimball commented on MAPREDUCE-1126:
------------------------------------------

To allow Avro-based comparators during the shuffle, a few changes need to take place. Here's my survey of the scope of the problem and proposal for addressing it.

HADOOP-6323 allows {{SerializationBase}} to provide {{RawComparator}} instances. But for Avro to be germane here, the {{Serializer}} and {{Deserializer}} instances used by {{MapOutputBuffer}} and other shuffle code need to be updated as well. {{MapOutputBuffer}} and other places where the {{Serializer}} and {{Deserializer}} interfaces are used should be changed to use instead {{SerializerBase}} and {{DeserializerBase}}. This should be done by changing the current calls to {{SerializationBase.getSerializer(keyClassName)}} to use the newer {{getSerializer(metadata)}} method.

Currently the only metadata we store is the {{Serialization-Class}} that identifies the intermediate key and value classes. This is implicitly set in, e.g. {{Job.setMapOutputKeyClass()}}. (It actually just stores a class name, and the {{getSerializer(className)}} method converts that to the appropriate metadata map for {{getSerializer(metadata)}}.) HADOOP-6165 allows an arbitrary map of metadata to be passed to the serialization framework. We should take advantange of this map throughout the shuffling process. The job's {{Configuration}} should be updated to embed a map a la the outline in HADOOP-6165 (See Tom's comment, part 4(ii)) to include a "family" of configuration parameters. Given the renaming of various parameters, I propose to embed the metadata maps in {{mapreduce.map.output.key.metadata}} and {{mapreduce.map.output.value.metadata}}.

The {{JobContext.setMapOutputKeyClass()}} method would instead of setting {{mapreduce.map.output.key.class}}, set the appropriate metadata strings under {{mapreduce.map.output.key.metadata}} or {{mapreduce.map.output.value.metadata}} to use the {{WritableSerialization}} implementation. If this metadata is not present, metadata could be inferred from {{mapreduce.map.output.key/value.class}}, which should be marked as deprecated.

{{JobContext}} would also include new methods which could be used instead of {{setMapOutputKeyClass()}}, e.g. {{setMapOutputKeySchema()}} which would accept an Avro schema and configure the shuffle to use AvroGenericSerialization for this data. A generic {{setMapOutputKeySerializationMetadata()}} could be added as well, which would accept any {{string -> string}} map of metadata for use with other arbitrary {{SerializationBase}} implementations. (It might be used internally by {{setMapOutputKeyClass()}} and {{setMapOutputKeySchema()}}.)

On the internal side, {{JobContext}} already provides a method {{getSortComparator()}} which provides the appropriate {{RawComparator}} implementation. But it does not directly provide {{SerializerBase}} or {{DeserializerBase}} instances; it instead only returns the key and value class names and clients of this (e.g., {{MapOutputBuffer}}) are expected to instantiate the correct serialization objects themselves. I think that {{JobContext}} should provide {{getMapOutputKeySerializer()}} and {{getMapOutputKeyDeserializer()}} methods that correctly handle either the use of {{mapreduce.map.output.key.metadata}} or {{mapreduce.map.output.key.class}} to avoid duplication of this logic.

All {{SerializerBase}}, {{DeserializerBase}}, and {{RawComparator}} implementations instantiated during shuffle would then be created via uses of this new API.

Longer term, metadata maps are necessary for reducer output as well; e.g., {{mapreduce.reduce.output.key.metadata}}. This would allow full end-to-end Avro use for user data. Similar methods for JobContext would need to be added to set these parameters, and {{Job.setOutputKeyClass()}} et al. would need to be updated to use this new API as well. But I think that this may be more reasonably part of MAPREDUCE-815 than the current issue, which is concerned with just the shuffle process.


> shuffle should use serialization to get comparator
> --------------------------------------------------
>
>                 Key: MAPREDUCE-1126
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task
>            Reporter: Doug Cutting
>
> Currently the key comparator is defined as a Java class.  Instead we should use the Serialization API to create key comparators.  This would permit, e.g., Avro-based comparators to be used, permitting efficient sorting of complex data types without having to write a RawComparator in Java.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.