hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-1295) Improve namenode restart times by short-circuiting the first block reports from datanodes
Date Thu, 21 Apr 2011 08:19:05 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matt Foley updated HDFS-1295:
-----------------------------

    Attachment: IBR_shortcut_v7atrunk.patch

The TestDatanodeBlockScanner and DFSTestUtil mods have been moved to bugs HDFS-1855, HDFS-1856,
and HDFS-1854.  Suresh's suggestions #1 and #2 regarding TestDatanodeBlockScanner were incorporated.

#3: Yes, the 20sec timeout is plenty long, because the config params were modified to make
the testcase run much faster.

#4: In block report time, the report creation time is included in metrics; see line 3156 of
FSNamesystem.  However, it is still useful to provide the log message as given, because the
metrics are bucketed and not possible to cross-correlate with a particular datanode.  Also,
the original log message (line 3137 of the unmodified file) stated that a report from a given
datanode was processed but didn't give the processing time.


> Improve namenode restart times by short-circuiting the first block reports from datanodes
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-1295
>                 URL: https://issues.apache.org/jira/browse/HDFS-1295
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: IBR_shortcut_v2a.patch, IBR_shortcut_v3atrunk.patch, IBR_shortcut_v4atrunk.patch,
IBR_shortcut_v4atrunk.patch, IBR_shortcut_v4atrunk.patch, IBR_shortcut_v6atrunk.patch, IBR_shortcut_v7atrunk.patch,
shortCircuitBlockReport_1.txt
>
>
> The namenode restart is dominated by the performance of processing block reports. On
a 2000 node cluster with 90 million blocks,  block report processing takes 30 to 40 minutes.
The namenode "diffs" the contents of the incoming block report with the contents of the blocks
map, and then applies these diffs to the blocksMap, but in reality there is no need to compute
the "diff" because this is the first block report from the datanode.
> This code change improves block report processing time by 300%.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message