hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Putu Yuwono (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-782) dynamic replication
Date Tue, 20 Nov 2012 08:22:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500914#comment-13500914

Putu Yuwono commented on HDFS-782:

I have several question for you guys who are more expert than me
1.What should be considered during the dynamic replicating process (ex:Block Usage, Cluster
Size)? How many new replicas should be created?
2.How to communicate between data nodes without bothering the name node?
3.How to find the best candidate for placing a new replica? What should be considered for
choosing the data node?
4.How to decide which one is better, moving or replicating? Is it depends on the usage pattern?

> dynamic replication
> -------------------
>                 Key: HDFS-782
>                 URL: https://issues.apache.org/jira/browse/HDFS-782
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Ning Zhang
> In a large and busy cluster, a block can be requested by many clients at the same time.
HDFS-767 tries to solve the failing case when the # of retries exceeds the maximum # of retries.
However, that patch doesn't solve the performance issue since all failing clients have to
wait a certain period before retry, and the # of retries could be high. 
> One solution to solve the performance issue is to increase the # of replicas for this
"hot" block dynamically when it is requested many times at a short period. The name node need
to be aware such situation and only clean up extra replicas when they are not accessed recently.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message