hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1295) Improve namenode restart times by short-circuiting the first block reports from datanodes
Date Tue, 12 Apr 2011 05:19:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018724#comment-13018724

Matt Foley commented on HDFS-1295:

Response to Hudson QA auto-test of 12/Apr/11 00:14 (PreCommit-HDFS-Build/346):

Of the four failing tests, two are our old friends hdfsproxy, unrelated to this Jira.
One (TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransferVerySmallWrite) is
three days old, and unrelated to this Jira.

The fourth, TestDatanodeBlockScanner.testTruncatedBlockReport, needs to be investigated, but
is likely to also be a test issue rather than a bug in the patch.
Also, I found that the patch for HDFS-1829 should be modified to use readLock() instead of
These are likely to be small changes, while the main patch to BlockManager is fairly large,
and likely to be unchanged by the fix to TestDatanodeBlockScanner.testTruncatedBlockReport.

Therefore, please consider starting code review if you are so inclined, so that we can complete
this submission soon.  Thank you very much.

> Improve namenode restart times by short-circuiting the first block reports from datanodes
> -----------------------------------------------------------------------------------------
>                 Key: HDFS-1295
>                 URL: https://issues.apache.org/jira/browse/HDFS-1295
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>         Attachments: IBR_shortcut_v2a.patch, IBR_shortcut_v3atrunk.patch, IBR_shortcut_v4atrunk.patch,
IBR_shortcut_v4atrunk.patch, IBR_shortcut_v4atrunk.patch, shortCircuitBlockReport_1.txt
> The namenode restart is dominated by the performance of processing block reports. On
a 2000 node cluster with 90 million blocks,  block report processing takes 30 to 40 minutes.
The namenode "diffs" the contents of the incoming block report with the contents of the blocks
map, and then applies these diffs to the blocksMap, but in reality there is no need to compute
the "diff" because this is the first block report from the datanode.
> This code change improves block report processing time by 300%.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message