hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1220) block not found in blockMap
Date Wed, 25 Apr 2007 18:44:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491727
] 

dhruba borthakur commented on HADOOP-1220:
------------------------------------------

This problem occured again. This time the block in question does not appear in the dfs data
directory. The namenode is requesting a block to be deleted. The datanode tries this operation
and encounters an error because the block is not in the blockMap. The processCommand() method
raises an exception. The code is such that the variable lastBlockReport is not set if processCommand()
raises an exception. This means that the datanode immediately send another block report to
the namenode.

In short, the above condition causes the datanode to send blockReports almost once every second!

I propose that we do the following:

1. in Datanode.offerService, replace the following piece of code 

          DatanodeCommand cmd = namenode.blockReport(dnRegistration,
                                                     data.getBlockReport());
          processCommand(cmd);
          lastBlockReport = now;


with

          DatanodeCommand cmd = namenode.blockReport(dnRegistration,
                                                     data.getBlockReport());
          lastBlockReport = now;
          processCommand(cmd);
          
2. In FSDataSet.invalidate:
    a) continue to process all blocks in invalidBlks[] even if one in the middle encounters
a problem.
    b) if getFile() returns null, still invoke volumeMap.get() and print whether we found
the block in 
       volumes or not. The volumeMap is used to generate the blockReport and this might help
in debugging.







> block not found in blockMap
> ---------------------------
>
>                 Key: HADOOP-1220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1220
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Koji Noguchi
>
> From Hadoop-973  debug message, we had datanode constantly printing out the following
message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected
error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block blk_-8672111663356339464
to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Unexpected
error trying to delete block blk_-3400783150525166031. Block not found in blockMap.
>   at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
>   at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
>   at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
>   at java.lang.Thread.run(Thread.java:619)
> ....
>  Ten minutes before these logs, there was 
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block blk_-3400783150525166031
> There was a  file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message