hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-803) Reducing memory consumption on Namenode : Part 1
Date Sat, 03 Feb 2007 00:29:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469918

Hairong Kuang commented on HADOOP-803:

> If the blocks were created after Namenode is started, one in inode will share the object
with blockMap. When name nodes starts up, it initially creates all the blocks in Indode while
reading the image file.. does not seem easy to share that reference. 

Even after a name node starts up, a block in inode does not share the object with blockMap.
So a name node contains two Block instantiations per block.

To reduce the number of block instantiations to 1, I think we can create a TreeSet of blocks
which maps a block id to its block object. When a block is intially created, simply add the
block to this block set. When later a reference to the block is needed, we can get it from
the block set.

We can furthur remove the blockMap and activeBlock list by adding two non-persistent fields
to the Block data structures. One is a reference to all its containing data nodes and one
to the inode representing the file that the block belongs to.

> Reducing memory consumption on Namenode : Part 1
> ------------------------------------------------
>                 Key: HADOOP-803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-803
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Raghu Angadi
>         Assigned To: Raghu Angadi
>             Fix For: 0.12.0
>         Attachments: block-refs-2.patch, block-refs-3.patch, block-refs-5.patch, HADOOP-803-2.patch,
HADOOP-803.patch, HADOOP-803_3.patch, HADOOP-803_4.patch, NameNodeMemoryHogs.txt
> There appears to be some places in Namenode that allow reducing memory consumption without
intrusive code or feature changes. This bug is an initial attempt making those changes. Please
include your thoughts as well. 
> One change I am planning to make : 
> Currently one copy of each block exists for each of the replicas and one copy for blockMap.
I think they are all supposed to be same.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message