hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
Date Thu, 09 Jul 2009 00:21:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728974#action_12728974

Sanjay Radia commented on HDFS-385:

  what is the usage model? Is it  along the following lines:

Say I start with policy 1 plugged into NN and Balancer. 

Later I change to policy 2 for both NN and Balancer.
Newly created files will follow policy 2.
Existing files will retain policy 1. Fsck will report violations for policy 2 for the old
files; correct?
If one starts the balancer (with policy 2) then it will move blocks around so that old files
match policy 2.

It would be an admin error to configure NN and Balancer with different policies; correct?
There is no check for this; correct?

Q. The policy manager is global to the file system. Can  it  have its own config to to do
different policies for different subtrees?

It would be useful to get the usage model documented.

> Design a pluggable interface to place replicas of blocks in HDFS
> ----------------------------------------------------------------
>                 Key: HDFS-385
>                 URL: https://issues.apache.org/jira/browse/HDFS-385
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>         Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt,
BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt
> The current HDFS code typically places one replica on local rack, the second replica
on remote random rack and the third replica on a random node of that remote rack. This algorithm
is baked in the NameNode's code. It would be nice to make the block placement algorithm a
pluggable interface. This will allow experimentation of different placement algorithms based
on workloads, availability guarantees and failure models.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message