hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1126) shuffle should use serialization to get comparator
Date Mon, 08 Feb 2010 17:42:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831013#action_12831013
] 

Doug Cutting commented on MAPREDUCE-1126:
-----------------------------------------

> Tom's patch adds two calls to that exact method.

One call is in a test method called getWritableSerializationMap, clearly serializaton-specific
code.  The other is in reading input splits, which has not been converted to use the new serialization
API, and still uses the split's class alone to determine serialization.

A primary goal of this issue is to update the shuffle to use the new Map<String,String>
means of naming serializations, so that any serialzation may be used, not just those determinable
from the class of the instance to be serialized.  Your patch uses the map output key class
alone to determine the serialization, and hence does not meet this goal.


> shuffle should use serialization to get comparator
> --------------------------------------------------
>
>                 Key: MAPREDUCE-1126
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task
>            Reporter: Doug Cutting
>            Assignee: Aaron Kimball
>             Fix For: 0.22.0
>
>         Attachments: m-1126-2.patch, m-1126-3.patch, MAPREDUCE-1126.2.patch, MAPREDUCE-1126.3.patch,
MAPREDUCE-1126.4.patch, MAPREDUCE-1126.5.patch, MAPREDUCE-1126.6.patch, MAPREDUCE-1126.patch,
MAPREDUCE-1126.patch
>
>
> Currently the key comparator is defined as a Java class.  Instead we should use the Serialization
API to create key comparators.  This would permit, e.g., Avro-based comparators to be used,
permitting efficient sorting of complex data types without having to write a RawComparator
in Java.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message