hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amir Langer (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-6658) Namenode memory optimization - Block replicas list
Date Thu, 10 Jul 2014 11:09:04 GMT
Amir Langer created HDFS-6658:

             Summary: Namenode memory optimization - Block replicas list 
                 Key: HDFS-6658
                 URL: https://issues.apache.org/jira/browse/HDFS-6658
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode
    Affects Versions: 2.4.1
            Reporter: Amir Langer
             Fix For: 3.0.0, 2.5.0

Part of the memory consumed by every BlockInfo object in the Namenode is a linked list of
block references for every DatanodeStorageInfo (called "triplets"). 
We propose to change the way we store the list in memory. 
Using primitive integer indexes instead of object references will reduce the memory needed
for every block replica (when compressed oops is disabled) and in our new design the list
overhead will be per DatanodeStorageInfo and not per block replica.

see attached design doc. for details and evaluation results.

This message was sent by Atlassian JIRA

View raw message