hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5724) Datanode should report deletion of blocks to Namenode explicitly
Date Tue, 12 May 2009 17:15:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708513#action_12708513

Suresh Srinivas commented on HADOOP-5724:

bq. I don't think so... the blocks are removed from the datanode list only after the blockreport.
from what I can see, could be mistaken.
Block is indeed deleted from the blocksMap when the file is deleted. The reason why it works
in this case is - if the next block report from the datanode includes the deleted block, since
the corresponding file is not found, the block is not added to the blocksMap.

To summarize the difference:
When a file is deleted, blocks can be deleted from blocksMap. Non existence file serves as
a reminder that the block is no longer valid. However when a block gets deleted from a datanode
(and the file still exists), there is nothing on the namenode that indicates that the block
has been deleted from the datanode.

> Datanode should report deletion of blocks to Namenode explicitly
> ----------------------------------------------------------------
>                 Key: HADOOP-5724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5724
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.21.0
>         Attachments: blockdel.patch, blockdel.patch
> Currently datanode notifies namenode newly added blocks and the blocks that are corrupt.
There is no explicit message from the datanode to the namenode to indicate the deletion of
blocks. Block reports from the datanode is the only way for the namenode to learn about the
deletion of blocks at a datanode. With the addition of explicit request to indicate to block
deletion, block report interval (which is currently 1 hour) can be increased to a longer duration.
This reduces load on both namenode and datanodes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message