hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-764) The memory consumption of processReport() in the namenode can be reduced
Date Thu, 07 Dec 2006 01:18:22 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-764?page=comments#action_12456276 ] 
            
Raghu Angadi commented on HADOOP-764:
-------------------------------------


As a related note, since we do a blocks.clear() in DatanodeDescriptor when we update blocks
from , this results in one separate  copy of a block exists for each node (i.e. one for each
replica) and one coy in NameNode's blockMap(). Ideally blocks in DatanodeDescriptor should
be references to blocks in global blockMap.

This patch decreases number of times blocks.clear() is invoked but over time there will be
separate copies of blocks. Fix is not to call blocks.clear() but update blocks map inline
when new blocks are removed or added inside processReport().
 


> The memory consumption of processReport() in the namenode can be reduced
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-764
>                 URL: http://issues.apache.org/jira/browse/HADOOP-764
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>         Attachments: processBlockReport3.patch
>
>
> The FSNamesystem.processReport() method converts the blocklist for a datanode into an
array by calling node.getBlocks(). Although this memory allocation is transient, it could
possibly require the garbage-collector to work that much harder. 
> The method Block.getBlocks() should be deprecated. Code that currently uses this method
should instead iterate over the Collection.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message