hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "omkar vinit joshi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-782) dynamic replication
Date Wed, 12 Dec 2012 06:57:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529678#comment-13529678
] 

omkar vinit joshi commented on HDFS-782:
----------------------------------------

Hi Putu,

This we have just implemented for hadoop 0.20.2 version. We have implemented 2 way approach.

1) We are changing the list of replicas returned by namenode ( based on xceivercount). Probability
of a particular node getting selected is inversely proportional to its active connections.
2) We are increasing and decreasing the replicas as needed. ( only when we see that all the
nodes hosting a block are loaded - hot spot).
At present we have kept various parameters such as maximum blocks to be replicated dynamically
( there by maximum extra space to be used) , number of extra replicas to create ( at present
2) configurable.
                
> dynamic replication
> -------------------
>
>                 Key: HDFS-782
>                 URL: https://issues.apache.org/jira/browse/HDFS-782
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Ning Zhang
>
> In a large and busy cluster, a block can be requested by many clients at the same time.
HDFS-767 tries to solve the failing case when the # of retries exceeds the maximum # of retries.
However, that patch doesn't solve the performance issue since all failing clients have to
wait a certain period before retry, and the # of retries could be high. 
> One solution to solve the performance issue is to increase the # of replicas for this
"hot" block dynamically when it is requested many times at a short period. The name node need
to be aware such situation and only clean up extra replicas when they are not accessed recently.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message