hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1444) Block allocation method does not check pendingCreates for duplicate block ids
Date Thu, 07 Jun 2007 20:30:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12502506

Konstantin Shvachko commented on HADOOP-1444:

- Redundant import:
import org.apache.hadoop.util.*;
- Redundant member:
private Log LOG = null;
- Member pendingCreates lost the comment explaining the meaning and the mapping type.

- Member pendingCreates comment should be updated.
- FSNamesystem.abandonBlock()
-- First stateChangeLog() report is redundant because it is already reported  by the NameNode.abandonBlock()
-- It would be better to report the result of block removal whether it was successful or not,
rather than reporting just the success and keeping it silent in case of failure.

- FSNamesystem.allocateBlock()
Placing pendingCreates.addBlock() inside the do-while-loop is confusing. Because you add block
just once when it is valid.
I'd propose to modify isValidBlock(b) so that it checked pendingCreates.contains(b) in addition
to checking whether it is in the blocksMap.
Then pendingCreates.addBlock() will need only to assert that the block is not pending.

> Block allocation method does not check pendingCreates for duplicate block ids
> -----------------------------------------------------------------------------
>                 Key: HADOOP-1444
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1444
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: duplicateBlockId.patch
> The HDFS namenode allocates a new random blockid when requested. It then checks the blocksMap
to verify if this blockid is already in use. If this block is is already in use, it generates
another random number and above process continues. When a blocksid that does not exist in
the blocksMap is found, it stores this blocksid in pendingCreateBlocks and returns the blocksid
to the requesting client.
> The above check for detecting duplicate blockid should check pendingCreateBlocks as well.
> A related problem exists when a file is deleted. Deleting a file causes all its blocks
to be deleted from the blocksMap immediately. These blockids move to recentInvalidateSets
and are sent out to the corresponding datanodes as part of responses to succeeding heartbeats.
So, there is a time window when a block exists in the datanode but not in the blocksMap. At
this time, if the blockid-random-number generator generates a blockid that exists in the datanode
but not on the blocksMap, then the namenode will fail to detect that this is a duplicate blockid.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message