hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1570) Shuffle stage - Key and Group Comparators
Date Tue, 30 Mar 2010 00:03:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851171#action_12851171
] 

Aaron Kimball commented on MAPREDUCE-1570:
------------------------------------------

Looks pretty good. The test failure is unrelated.

Some comments on the implementation in src/contrib/mrunit/src/java/org/apache/hadoop/mrunit/MapReduceDriverBase.java:

{code}
+    keyValueOrderComparator = simpleComparator;
{code}

Are these ordinarily ordered by default? I thought they were placed in an undefined ordering.
Does this change the semantics of the programs under test to enforce here? I think that the
keyValueOrderComparator should remain null, and in in shuffle() step 2, if this comparator
is null, the secondary sort is just skipped.

{code}
+        groupedKeyList = new Vector<Pair<K2,V2>>();
{code}

Don't use Vector; use ArrayList instead unless you specifically need thread safety (I don't
think you do here).


> Shuffle stage - Key and Group Comparators
> -----------------------------------------
>
>                 Key: MAPREDUCE-1570
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1570
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/mrunit
>            Reporter: Chris White
>            Assignee: Chris White
>            Priority: Minor
>         Attachments: MAPREDUCE-1570.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Shuffle method in org.apache.hadoop.mrunit.MapReduceDriverBase doesn't currently allow
the use of custom GroupingComparator and SortComparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message