hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7891) A block placement policy with best rack failure tolerance
Date Thu, 16 Apr 2015 01:25:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497428#comment-14497428
] 

Walter Su commented on HDFS-7891:
---------------------------------

bq. If we simply let getMaxNodesPerRack() return the ceiling of totalNumOfReplicas/numOfRacks,
like (totalNumOfReplicas - 1) / numOfRacks + 1, then do not override chooseTargetInOrder(),
can we get the right distribution we want?
If totalNumOfReplicas < numOfRacks, it's much simpler, maxNodesPerRack==1. It's usually
the case in production cluster. The tricky thing is, when totalNumOfReplicas > numOfRacks.
*Notation*
R = total racks in cluster
X = total expected replicas
Q=Math.floor(X/R)
T=X%R
if X<R, X racks have 1 replica.
if X>R && T==0, R racks have Q replicas.
if X>R && T!=0, X-T racks have Q replicas. T racks have Q+1 replicas.

The tricky thing is, 
1. if X>R && T!=0, X-T racks have Q replicas. T racks have Q+1 replicas. *We want
local rack in the T racks group*. So we have to override chooseTargetInOrder() anyway.
2. if X>R && T!=0, and if _"we simply let getMaxNodesPerRack() return the ceiling
of totalNumOfReplicas/numOfRacks"_, there is a chance that more than T racks have Q+1 replicas
while some racks have none.

> A block placement policy with best rack failure tolerance
> ---------------------------------------------------------
>
>                 Key: HDFS-7891
>                 URL: https://issues.apache.org/jira/browse/HDFS-7891
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: namenode
>            Reporter: Walter Su
>            Assignee: Walter Su
>            Priority: Minor
>         Attachments: HDFS-7891.005.dup.patch, HDFS-7891.005.patch
>
>
> a block placement policy tries its best to place replicas to most racks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message