hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss
Date Sat, 29 May 2010 05:18:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873238#action_12873238

dhruba borthakur commented on HDFS-1094:

>From all the above mathematical formulae, can we then say that the policy listed below
can be a good first step:

"The first replica is on a node on rack r. Then the other two replicas be randomly selected
nodes on either rack r-1 or r+1. In this approach, three datanodes in two consecutive racks
have to fail simultaneously for a block loss to occur. This is better than the current implementation
where any three datanode failures in the entire cluster can cause one block to be lost."

The NetworkTopology class in the namenode already has information about node to rack location.

> Intelligent block placement policy to decrease probability of block loss
> ------------------------------------------------------------------------
>                 Key: HDFS-1094
>                 URL: https://issues.apache.org/jira/browse/HDFS-1094
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: Rodrigo Schmidt
>         Attachments: prob.pdf, prob.pdf
> The current HDFS implementation specifies that the first replica is local and the other
two replicas are on any two random nodes on a random remote rack. This means that if any three
datanodes die together, then there is a non-trivial probability of losing at least one block
in the cluster. This JIRA is to discuss if there is a better algorithm that can lower probability
of losing a block.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message