hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-2537) re-replicating under replicated blocks should be more dynamic
Date Thu, 03 Nov 2011 19:45:32 GMT
re-replicating under replicated blocks should be more dynamic
-------------------------------------------------------------

                 Key: HDFS-2537
                 URL: https://issues.apache.org/jira/browse/HDFS-2537
             Project: Hadoop HDFS
          Issue Type: Improvement
    Affects Versions: 0.20.205.0, 0.23.0
            Reporter: Nathan Roberts


When a node fails or is decommissioned, a large number of blocks become under-replicated.
Since re-replication work is distributed, the hope would be that all blocks could be restored
to their desired replication factor in very short order. This doesn't happen though because
the load the cluster is willing to devote to this activity is mostly static (controlled by
configuration variables). Since it's mostly static, the rate has to be set conservatively
to avoid overloading the cluster with replication work.

This problem is especially noticeable when you have lots of small blocks. It can take many
hours to re-replicate the blocks that were on a node while the cluster is mostly idle. 


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message