hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2537) re-replicating under replicated blocks should be more dynamic
Date Mon, 07 Nov 2011 17:10:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145621#comment-13145621
] 

Aaron T. Myers commented on HDFS-2537:
--------------------------------------

I agree with all of this, Nathan. I don't have time to work on this right now, but will be
happy to help with reviews if someone else does have time to devote to this issue.
                
> re-replicating under replicated blocks should be more dynamic
> -------------------------------------------------------------
>
>                 Key: HDFS-2537
>                 URL: https://issues.apache.org/jira/browse/HDFS-2537
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 0.20.205.0, 0.23.0
>            Reporter: Nathan Roberts
>
> When a node fails or is decommissioned, a large number of blocks become under-replicated.
Since re-replication work is distributed, the hope would be that all blocks could be restored
to their desired replication factor in very short order. This doesn't happen though because
the load the cluster is willing to devote to this activity is mostly static (controlled by
configuration variables). Since it's mostly static, the rate has to be set conservatively
to avoid overloading the cluster with replication work.
> This problem is especially noticeable when you have lots of small blocks. It can take
many hours to re-replicate the blocks that were on a node while the cluster is mostly idle.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message