hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-803) Reducing memory consumption on Namenode : Part 1
Date Sat, 03 Feb 2007 01:14:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469924
] 

Raghu Angadi commented on HADOOP-803:
-------------------------------------


> Even after a name node starts up, a block in inode does not share the object with blockMap.
> So a name node contains two Block instantiations per block.

In the patch, when a file is closed, we do use the reference in blockMap (because, datanode
that has block would have informed namenode already).

Yes, we can get rid of one of blockMap and activeBlocks maps (not both). This also removes
extra block object we have for files that exist before namenode restarts. The changes for
this are a bit more intrusive, I am wondering if I should do it as part of this patch..




> Reducing memory consumption on Namenode : Part 1
> ------------------------------------------------
>
>                 Key: HADOOP-803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-803
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Raghu Angadi
>         Assigned To: Raghu Angadi
>             Fix For: 0.12.0
>
>         Attachments: block-refs-2.patch, block-refs-3.patch, block-refs-5.patch, HADOOP-803-2.patch,
HADOOP-803.patch, HADOOP-803_3.patch, HADOOP-803_4.patch, NameNodeMemoryHogs.txt
>
>
> There appears to be some places in Namenode that allow reducing memory consumption without
intrusive code or feature changes. This bug is an initial attempt making those changes. Please
include your thoughts as well. 
> One change I am planning to make : 
> Currently one copy of each block exists for each of the replicas and one copy for blockMap.
I think they are all supposed to be same.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message