hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-803) Reducing memory consumption on Namenode : Part 1
Date Fri, 02 Feb 2007 00:37:05 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Raghu Angadi updated HADOOP-803:
--------------------------------

    Attachment: NameNodeMemoryHogs.txt

see attached NameNodeMemoryHogs.txt for better formated list. Following is cut-n-paste of
the attachment.

Main FS wide datastructures : 
----------------------------- 
    
    blockMap        : block --> containing nodes : HashMap  
    activeBLocks    : block --> INode : HashMap 
    INode.children  : FileName --> INode : TreeMap (in dir INode). 
    DN.blocks       : block --> self : TreeMap ( in each DataNodeDescriptor ) 
    datanodeMap     : String --> DatanodeDescriptor ( linear cost with 
                                                      number of dataNodes ) 
Memory cost per Block : 
---------------------- 
  Before this patch : 
     Block Objects : 3 ( one per replica in DN.blocks) 
                     1 ( in blockMap ) 
                     1 ( in INode ); 
     TreeMap       : 1 ( containingNode TreeSet )  
     TreeMap$Entry : 3 ( conatiningNodes TreeSet in blockMap ) 
                     3 ( in DN.blocks in each of the nodes ) 
     HashMap$Entry : 1 ( blockMap ) 
                     1 ( activeBlocks ) 
 
  After this patch : 
     Block Objects : 1 ( in blockMap ) 
                     1 ( in INode that existed before NameNode restarted ) 
     ArrayList     : 1 ( containingNodes Arraylist )  
     TreeMap$Entry : 3 ( in DN.blocks in each of the nodes ) 
     HashMap$Entry : 1 ( blockMap ) 
                     1 ( activeBlocks ) 
 
Memory cost per INode : 
---------------------- 
     TreeMap       : 1 ( children. removed in this issue for regular files ) 
     TreeMap$Entry : 1 ( in parent.children ) 
     INode object  : 1 
     char[128]     : 1 (char[] used in String) 
     String        : 1 ( name ) 


> Reducing memory consumption on Namenode : Part 1
> ------------------------------------------------
>
>                 Key: HADOOP-803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-803
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Raghu Angadi
>         Assigned To: Raghu Angadi
>             Fix For: 0.11.0
>
>         Attachments: block-refs-2.patch, block-refs-3.patch, block-refs-5.patch, HADOOP-803-2.patch,
HADOOP-803.patch, HADOOP-803_3.patch, NameNodeMemoryHogs.txt
>
>
> There appears to be some places in Namenode that allow reducing memory consumption without
intrusive code or feature changes. This bug is an initial attempt making those changes. Please
include your thoughts as well. 
> One change I am planning to make : 
> Currently one copy of each block exists for each of the replicas and one copy for blockMap.
I think they are all supposed to be same.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message