Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Wed, 8 Jun 2011 22:21:58 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <1725676215.5028.1307571718954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HDFS-1295) Improve namenode restart times by
 short-circuiting the first block reports from datanodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HDFS-1295:
-----------------------------

    Attachment: HDFS-1295_delta_for_trunk.patch

Delta patch to re-integrate HDFS-1295 with DataNode.java in trunk.

> Improve namenode restart times by short-circuiting the first block reports from datanodes
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-1295
>                 URL: https://issues.apache.org/jira/browse/HDFS-1295
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.23.0
>            Reporter: dhruba borthakur
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1295_delta_for_trunk.patch, IBR_shortcut_v2a.patch, IBR_shortcut_v3atrunk.patch, IBR_shortcut_v4atrunk.patch, IBR_shortcut_v4atrunk.patch, IBR_shortcut_v4atrunk.patch, IBR_shortcut_v6atrunk.patch, IBR_shortcut_v7atrunk.patch, shortCircuitBlockReport_1.txt
>
>
> The namenode restart is dominated by the performance of processing block reports. On a 2000 node cluster with 90 million blocks,  block report processing takes 30 to 40 minutes. The namenode "diffs" the contents of the incoming block report with the contents of the blocks map, and then applies these diffs to the blocksMap, but in reality there is no need to compute the "diff" because this is the first block report from the datanode.
> This code change improves block report processing time by 300%.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira