hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
Date Thu, 09 Jul 2009 23:38:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729453#action_12729453

dhruba borthakur commented on HDFS-385:

I agree that that this API might have to evolve over time. We should mark it as "unstable"
in bold letters.

> In the past folks have complained that hadoop is too easy to misconfigure.

The default policy should work well for 99/9% people out there. Only a system admin can change
the default policy. And one has to write Java code to implement a new policy... making it
even tougher for most people to change policy.

>Given the above should the system record the policy in the fsImage to prevent it from
being changed? Similarly should the balancer check to see if it has the same policy as the
This can be done. This is mostly to reduce configuration errors, right? If so, can we defer
it till we see it being a problem?

> However the experimentation is useful and as long it does not impact the base code in
a negative way, we should be able to add such features to hadoop after careful review.

Thanks. Please review the code and provide some feedback if you so desire.

> Design a pluggable interface to place replicas of blocks in HDFS
> ----------------------------------------------------------------
>                 Key: HDFS-385
>                 URL: https://issues.apache.org/jira/browse/HDFS-385
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>         Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt,
BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt
> The current HDFS code typically places one replica on local rack, the second replica
on remote random rack and the third replica on a random node of that remote rack. This algorithm
is baked in the NameNode's code. It would be nice to make the block placement algorithm a
pluggable interface. This will allow experimentation of different placement algorithms based
on workloads, availability guarantees and failure models.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message