hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "omkar vinit joshi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-782) dynamic replication
Date Wed, 12 Dec 2012 06:57:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529678#comment-13529678

omkar vinit joshi commented on HDFS-782:

Hi Putu,

This we have just implemented for hadoop 0.20.2 version. We have implemented 2 way approach.

1) We are changing the list of replicas returned by namenode ( based on xceivercount). Probability
of a particular node getting selected is inversely proportional to its active connections.
2) We are increasing and decreasing the replicas as needed. ( only when we see that all the
nodes hosting a block are loaded - hot spot).
At present we have kept various parameters such as maximum blocks to be replicated dynamically
( there by maximum extra space to be used) , number of extra replicas to create ( at present
2) configurable.
> dynamic replication
> -------------------
>                 Key: HDFS-782
>                 URL: https://issues.apache.org/jira/browse/HDFS-782
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Ning Zhang
> In a large and busy cluster, a block can be requested by many clients at the same time.
HDFS-767 tries to solve the failing case when the # of retries exceeds the maximum # of retries.
However, that patch doesn't solve the performance issue since all failing clients have to
wait a certain period before retry, and the # of retries could be high. 
> One solution to solve the performance issue is to increase the # of replicas for this
"hot" block dynamically when it is requested many times at a short period. The name node need
to be aware such situation and only clean up extra replicas when they are not accessed recently.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message