hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hui Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode
Date Tue, 24 Mar 2015 10:36:52 GMT
Hui Zheng created HDFS-7980:

             Summary: Incremental BlockReport will dramatically slow down the startup of 
a namenode
                 Key: HDFS-7980
                 URL: https://issues.apache.org/jira/browse/HDFS-7980
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Hui Zheng

In the current implementation the datanode will call the reportReceivedDeletedBlocks() method
that is a IncrementalBlockReport before calling the bpNamenode.blockReport() method. So in
a large(several thousands of datanodes) and busy cluster it will slow down(more than one hour)
the startup of namenode. 

List<DatanodeCommand> blockReport() throws IOException {
    // send block report if timer has expired.
    final long startTime = now();
    if (startTime - lastBlockReport <= dnConf.blockReportInterval) {
      return null;

    final ArrayList<DatanodeCommand> cmds = new ArrayList<DatanodeCommand>();

    // Flush any block information that precedes the block report. Otherwise
    // we have a chance that we will miss the delHint information
    // or we will report an RBW replica after the BlockReport already reports
    // a FINALIZED one.
    lastDeletedReport = startTime;
        // Send the reports to the NN.
    int numReportsSent = 0;
    int numRPCs = 0;
    boolean success = false;
    long brSendStartTime = now();
    try {
      if (totalBlockCount < dnConf.blockReportSplitThreshold) {
        // Below split threshold, send all reports in a single message.
        DatanodeCommand cmd = bpNamenode.blockReport(
            bpRegistration, bpos.getBlockPoolId(), reports);

This message was sent by Atlassian JIRA

View raw message