hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5724) Datanode should report deletion of blocks to Namenode explicitly
Date Tue, 12 May 2009 15:31:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708466#action_12708466

Raghu Angadi commented on HADOOP-5724:

One reason could be because if there is a block report that is sent by DN before seeing the
deletion request, and seen by NN after sending the deletion request, that block will be added
again to the datanode. There might be other related issues.  Unless these are all sorted out,
it is better for block list for a DN at NN to reflect what DN says it is. 

> But in case of block deletion due to file deletion etc., blocks are removed from blocksMap

I don't think so... the blocks are removed from the datanode list only after the blockreport.
from what I can see, could be mistaken.  Otherwise, when a file is deleted, a block could
be safely deleted from the datanode map. Does getstamp handle all the case where NN might
reuse the same block id before datanodes delete the prev incarnations?

Also the extra overhead for reporting the deleted blocks is more than compensated by less
frequent block reports.

> Datanode should report deletion of blocks to Namenode explicitly
> ----------------------------------------------------------------
>                 Key: HADOOP-5724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5724
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.21.0
>         Attachments: blockdel.patch, blockdel.patch
> Currently datanode notifies namenode newly added blocks and the blocks that are corrupt.
There is no explicit message from the datanode to the namenode to indicate the deletion of
blocks. Block reports from the datanode is the only way for the namenode to learn about the
deletion of blocks at a datanode. With the addition of explicit request to indicate to block
deletion, block report interval (which is currently 1 hour) can be increased to a longer duration.
This reduces load on both namenode and datanodes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message