hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1861) Raid should rearrange the replicas while raiding
Date Fri, 05 Nov 2010 05:50:44 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928484#action_12928484

Hadoop QA commented on MAPREDUCE-1861:

-1 overall.  Here are the results of testing the latest attachment 
  against trunk revision 967297.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit

    +1 core tests.  The patch passed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/console

This message is automatically generated.

> Raid should rearrange the replicas while raiding
> ------------------------------------------------
>                 Key: MAPREDUCE-1861
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1861
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/raid
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>         Attachments: MAPREDUCE-1861.txt, MAPREDUCE-1861.txt
> Raided file introduce extra dependencies on the blocks on the same stripe.
> Therefore we need a new way to place the blocks.
> It is desirable that raided file satisfies the following two conditions:
> a. Replicas on the same stripe should be on different machines (or racks)
> b. Replicas of the same block should be on different racks
> MAPREDUCE-1831 will try to delete the replicas on the same stripe and the same machine
> But in the mean time, it will try to maintain the number of distinct racks of one block
> We cannot satisfy (a) and (b) at the same time with the current logic in BlockPlacementPolicyDefault.chooseTarget().
> One choice we have is to change BlockPlacementPolicyDefault.chooseTarget().
> However, this placement is in general good for all files including the unraided ones.
> It is not clear to us that we can make this good for both raided and unraided files.
> So we propose this idea that when raiding the file. We create one more off-rack replica
(so the replication=4 now).
> Than we delete two blocks using the policy in MAPREDUCE-1831 after that (replication=2
> This way we can rearrange the replicas to satisfy (a) and (b) at the same time.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message