hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-4183) Throttle block recovery
Date Tue, 13 Nov 2012 02:15:14 GMT
Kihwal Lee created HDFS-4183:

             Summary: Throttle block recovery
                 Key: HDFS-4183
                 URL: https://issues.apache.org/jira/browse/HDFS-4183
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: name-node
    Affects Versions: 2.0.2-alpha, 0.23.4
            Reporter: Kihwal Lee
            Priority: Critical

When a large number of files are abandoned without closing, a storm of lease expiration follows
in about an hour (lease hard limit). For the last block of each file, block recovery is initiated
and when the datanode is done, it calls commitBlockSynchronization() is called against namenode.
A burst of these calls can slow down namenode considerably. We need to throttle block recovery
and/or speed up the rate at which commitBlockSynchronization() is served.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message