hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1295) Improve namenode restart times by short-circuiting the first block reports from datanodes
Date Thu, 21 Apr 2011 19:43:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022888#comment-13022888
] 

Hadoop QA commented on HDFS-1295:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12476974/IBR_shortcut_v7atrunk.patch
  against trunk revision 1095789.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit
warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.TestFileConcurrentReader

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/399//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/399//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/399//console

This message is automatically generated.

> Improve namenode restart times by short-circuiting the first block reports from datanodes
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-1295
>                 URL: https://issues.apache.org/jira/browse/HDFS-1295
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: IBR_shortcut_v2a.patch, IBR_shortcut_v3atrunk.patch, IBR_shortcut_v4atrunk.patch,
IBR_shortcut_v4atrunk.patch, IBR_shortcut_v4atrunk.patch, IBR_shortcut_v6atrunk.patch, IBR_shortcut_v7atrunk.patch,
shortCircuitBlockReport_1.txt
>
>
> The namenode restart is dominated by the performance of processing block reports. On
a 2000 node cluster with 90 million blocks,  block report processing takes 30 to 40 minutes.
The namenode "diffs" the contents of the incoming block report with the contents of the blocks
map, and then applies these diffs to the blocksMap, but in reality there is no need to compute
the "diff" because this is the first block report from the datanode.
> This code change improves block report processing time by 300%.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message