hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number of maps
Date Wed, 06 Jan 2010 08:59:54 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797034#action_12797034
] 

Hemanth Yamijala commented on HADOOP-6439:
------------------------------------------

This is looking good. I have a very minor comment. In the test case, I would recommend we
write a deprecated key with multiple values - like the key X, and make sure that loadResource
actually loads all the keys - both old and new. With this change, I think the patch is good
to go.

Can you please run tests with the new common jar file with both HDFS and MapReduce projects
and upload results of those also here. This is in addition to Hudson validation of the common
project's unit tests itself.

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch,
mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets
the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead,
use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use
mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message