hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-1295) Improve namenode restart times by short-circuiting the first block reports from datanodes
Date Mon, 11 Apr 2011 07:01:06 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matt Foley updated HDFS-1295:
-----------------------------

    Attachment: IBR_shortcut_v4atrunk.patch

Small change in IBR_shortcut to correctly adapt to use of isPopulatingReplQueues() in current
trunk.

Also included changes to the six unit tests that were failing.  Several timed out in Hudson
without useful error logs.  These I examined, added timeouts and throws of TimeoutException,
and useful log info; in some cases I also fixed what appeared by code inspection to be the
source of the problem.  They are likely to need another pass, but submitting them through
Hudson to see.  (None fail on my local test environment.)

> Improve namenode restart times by short-circuiting the first block reports from datanodes
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-1295
>                 URL: https://issues.apache.org/jira/browse/HDFS-1295
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.23.0
>
>         Attachments: IBR_shortcut_v2a.patch, IBR_shortcut_v3atrunk.patch, IBR_shortcut_v4atrunk.patch,
shortCircuitBlockReport_1.txt
>
>
> The namenode restart is dominated by the performance of processing block reports. On
a 2000 node cluster with 90 million blocks,  block report processing takes 30 to 40 minutes.
The namenode "diffs" the contents of the incoming block report with the contents of the blocks
map, and then applies these diffs to the blocksMap, but in reality there is no need to compute
the "diff" because this is the first block report from the datanode.
> This code change improves block report processing time by 300%.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message