hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-1295) Improve namenode restart times by short-circuiting the first block reports from datanodes
Date Mon, 12 Jul 2010 23:56:53 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

dhruba borthakur updated HDFS-1295:
-----------------------------------

    Attachment: shortCircuitBlockReport_1.txt

This patch applied only to 0.20 and does two things:

1. The first block report from a datanode is directly inserted into the blocksMap
2. Optimize the countLiveNodes methods to be lightweight

I extended the attached unit test to gather the average time to process the first block report
from a datanode:

||Configuration|                       without patch | with patch||
|40 nodes, 200K blocks each | 232 sec        | 80 sec |
|80 nodes, 200K blocks each|  240 sec        | 82 sec |


> Improve namenode restart times by short-circuiting the first block reports from datanodes
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-1295
>                 URL: https://issues.apache.org/jira/browse/HDFS-1295
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.22.0
>
>         Attachments: shortCircuitBlockReport_1.txt
>
>
> The namenode restart is dominated by the performance of processing block reports. On
a 2000 node cluster with 90 million blocks,  block report processing takes 30 to 40 minutes.
The namenode "diffs" the contents of the incoming block report with the contents of the blocks
map, and then applies these diffs to the blocksMap, but in reality there is no need to compute
the "diff" because this is the first block report from the datanode.
> This code change improves block report processing time by 300%.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message