hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Riza Suminto (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-13517) MetaSave command can block NameNode for long time
Date Tue, 01 May 2018 17:13:00 GMT
Riza Suminto created HDFS-13517:

             Summary: MetaSave command can block NameNode for long time
                 Key: HDFS-13517
                 URL: https://issues.apache.org/jira/browse/HDFS-13517
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode
    Affects Versions: 2.9.0
            Reporter: Riza Suminto

hdfs metasave command do full iterations over BlockManager list, such as neededReplications, postponedMisreplicatedBlocks,
and so on. This does not scale well when there are millions of under-replicated data blocks
in the cluster, due heavy load or network error.

We test this metasave command by modifying NNThroughputBenchmark to simulate large number
of under-replicated data blocks. We found that when there are about 16 millions under-replicated
blocks, metasave command can take up to 29 second while holding FNamesystem write lock. It
is probably safer to cap the iteration and output size of metasave command, so that it does
not block NameNode for too long.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message