hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1831) Delete the co-located replicas when raiding file
Date Sat, 19 Jun 2010 00:26:25 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880388#action_12880388

Scott Chen commented on MAPREDUCE-1831:

Rodrigo: Thanks for the comment. In
chooseReplicaToDelete(inode, block, replicationFactor, first, second);
We always go with the "first" set first. first set contains the replicas on the same rack.
In the normal case, the "local" replica will be in the second set.
In this patch, if we have a tie in the number of co-located replica, we compare the free space.
So this policy will just fall back to function as BlockPlacementPolicyDefault for the files
with replication=3.

To solve this problem we need to create one more replica (make the replication=4) while raiding.
It is described in MAPREDUCE-1861.
After we have another off-rack replica, this policy kicks in to reduce the replication to
This policy will first remove a replica in first set. And remove the replica co-located on
the same stripe.
The final result will be two replica on different rack and no co-located replica on the same

We need both this JIRA and MAPREDUCE-1861 to achieve that result.

> Delete the co-located replicas when raiding file
> ------------------------------------------------
>                 Key: MAPREDUCE-1831
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1831
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/raid
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>         Attachments: MAPREDUCE-1831.20100610.txt, MAPREDUCE-1831.txt, MAPREDUCE-1831.v1.1.txt
> In raid, it is good to have the blocks on the same stripe located on different machine.
> This way when one machine is down, it does not broke two blocks on the stripe.
> By doing this, we can decrease the block error probability in raid from O(p^3) to O(p^4)
which can be a hugh improvement (where p is the replica missing probability).
> One way to do this is that we can add a new BlockPlacementPolicy which deletes the replicas
that are co-located.
> So when raiding the file, we can make the remaining replicas live on different machines.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message