hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1297) datanode sending block reports to namenode once every second
Date Wed, 25 Apr 2007 21:26:15 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

dhruba borthakur updated HADOOP-1297:

    Attachment: datanodeDeleteBlocks.patch

> datanode sending block reports to namenode once every second
> ------------------------------------------------------------
>                 Key: HADOOP-1297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1297
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>         Attachments: datanodeDeleteBlocks.patch
> The namenode is requesting a block to be deleted. The datanode tries this operation and
encounters an error because the block is not in the blockMap. The processCommand() method
raises an exception. The code is such that the variable lastBlockReport is not set if processCommand()
raises an exception. This means that the datanode immediately send another block report to
the namenode. The eats up quite a bit of CPU on namenode.
> In short, the above condition causes the datanode to send blockReports almost once every
> I propose that we do the following:
> 1. in Datanode.offerService, replace the following piece of code
> DatanodeCommand cmd = namenode.blockReport(dnRegistration,
> data.getBlockReport());
> processCommand(cmd);
> lastBlockReport = now;
> with
> DatanodeCommand cmd = namenode.blockReport(dnRegistration,
> data.getBlockReport());
> lastBlockReport = now;
> processCommand(cmd);
> 2. In FSDataSet.invalidate:
> a) continue to process all blocks in invalidBlks[] even if one in the middle encounters
a problem.
> b) if getFile() returns null, still invoke volumeMap.get() and print whether we found
the block in
> volumes or not. The volumeMap is used to generate the blockReport and this might help
in debugging.
> [

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message