hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-1391) Exiting safemode takes a long time when there are lots of blocks in the HDFS
Date Tue, 10 May 2011 22:39:47 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Matt Foley updated HDFS-1391:

    Attachment: 1391_excessReplicas5_comments.txt

Hi Dhruba, thanks for doing this patch.  It's interesting to see how complex the refactoring
is to implement this conceptually straightforward enhancement!

Do you think there's an opportunity to do something similar with _under_ replicated block
processing? (in another Jira, of course)  Like the current patch, this would help avoid lengthy
delays in exiting safe mode at startup.

I've code reviewed the patch.  Comments were a little long so I attached them as a document.

> Exiting safemode takes a long time when there are lots of blocks in the HDFS
> ----------------------------------------------------------------------------
>                 Key: HDFS-1391
>                 URL: https://issues.apache.org/jira/browse/HDFS-1391
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: 1391_excessReplicas5_comments.txt, excessReplicas.1_trunk.txt, excessReplicas2.txt,
excessReplicas3.txt, excessReplicas5.txt
> When the namenode decides to exit safemode,  it acquires the FSNamesystem lock and then
iterates over all blocks in the blocksmap to determine if any block has any excess replicas.
This call takes upwards of 5 minutes on a cluster that has 100 million blocks. This delays
namenode restart to a good extent.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message