hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gokul (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1230) BlocksMap.blockinfo is not getting cleared immediately after deleting a block.This will be cleared only after block report comes from the datanode.Why we need to maintain the blockinfo till that time.
Date Wed, 23 Jun 2010 06:44:50 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881574#action_12881574
] 

Gokul commented on HDFS-1230:
-----------------------------


{quote}
Konstantin Shvachko added a comment - 22/Jun/10 10:36 PM
What is BlocksMap.blockinfo?
{quote}

FYI, BlockInfo is the static inner class of BlocksMap which is used for holding block metadata.

{quote}
If this is a question please use hdfs lists rather than creating jiras.
{quote}

Not a question. 

{quote}
If not please clarify what exactly you would like to improve. 
{quote}

Yup.! Here we go. 

Whenever a delete is done, the removePathAndBlocks() in FsNameSystem is internally called.

{noformat}
  void removePathAndBlocks(String src, List<Block> blocks) throws IOException {

		leaseManager.removeLeaseWithPrefixPath(src);
		for (Block b : blocks) {
			blocksMap.removeINode(b);
			corruptReplicas.removeFromCorruptReplicasMap(b);
			addToInvalidates(b);
		}
	}
{noformat}
			

Here the corresponding block is not removed by  blocksMap.removeINode(b) since it checks for
number of datanodes = 0 and then removing the enrty. 

so when exactly will it be removed from the map?? It will be removed during the next block
report of the datanode to the namenode which  by default is 1 hour. 

My concern here is why should we maintain the unwanted entry in the Map (*Map<BlockInfo,
BlockInfo> map*) in BlocksMap till that time??? We can clear the entry in removePathAndBlocks()
itself something like this right??
{noformat}
blocksMap.removeBlock(blocksMap.getStoredBlock(b));
{noformat}

I got this issue when I found that the NameNode's BlocksMap memory usage keeps on increasing
during intensive write/mkdir operations even if they are deleted. 


> BlocksMap.blockinfo is not getting cleared immediately after deleting a block.This will
be cleared only after block report comes from the datanode.Why we need to maintain the blockinfo
till that time.
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1230
>                 URL: https://issues.apache.org/jira/browse/HDFS-1230
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.20.1
>            Reporter: Gokul
>
> BlocksMap.blockinfo is not getting cleared immediately after deleting a block.This will
be cleared only after block report comes from the datanode.Why we need to maintain the blockinfo
till that time???? It increases namenode memory unnecessarily. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message