Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Date: Thu, 3 Nov 2011 19:45:32 +0000 (UTC)
From: "Nathan Roberts (Created) (JIRA)" <jira@apache.org>
To: hdfs-dev@hadoop.apache.org
Message-ID: 
 <802123958.57055.1320349532592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HDFS-2537) re-replicating under replicated blocks
 should be more dynamic
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

re-replicating under replicated blocks should be more dynamic
-------------------------------------------------------------

                 Key: HDFS-2537
                 URL: https://issues.apache.org/jira/browse/HDFS-2537
             Project: Hadoop HDFS
          Issue Type: Improvement
    Affects Versions: 0.20.205.0, 0.23.0
            Reporter: Nathan Roberts


When a node fails or is decommissioned, a large number of blocks become under-replicated. Since re-replication work is distributed, the hope would be that all blocks could be restored to their desired replication factor in very short order. This doesn't happen though because the load the cluster is willing to devote to this activity is mostly static (controlled by configuration variables). Since it's mostly static, the rate has to be set conservatively to avoid overloading the cluster with replication work.

This problem is especially noticeable when you have lots of small blocks. It can take many hours to re-replicate the blocks that were on a node while the cluster is mostly idle. 


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira