hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Bortnikov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list
Date Mon, 14 Jul 2014 20:48:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061213#comment-14061213
] 

Edward Bortnikov commented on HDFS-6658:
----------------------------------------

As opposed to block-management-as-a-service (HDFS-5477), this optimization is very scoped
(data structure modification), and introduces minimal risk. The saving is about 20% of block
management footprint, or about 10% of the total NN footprint.  

The design in HDFS-5477 details why off-heap swap space management is not an option in high-end
settings (terabytes of metadata). If the off-heap memory is managed on SSD, this is still
two orders of magnitude slower than DDR3. In this setting, block reports in large clusters
cannot be sustained because they have no locality of reference. 

> Namenode memory optimization - Block replicas list 
> ---------------------------------------------------
>
>                 Key: HDFS-6658
>                 URL: https://issues.apache.org/jira/browse/HDFS-6658
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.4.1
>            Reporter: Amir Langer
>            Assignee: Amir Langer
>         Attachments: Namenode Memory Optimizations - Block replicas list.docx
>
>
> Part of the memory consumed by every BlockInfo object in the Namenode is a linked list
of block references for every DatanodeStorageInfo (called "triplets"). 
> We propose to change the way we store the list in memory. 
> Using primitive integer indexes instead of object references will reduce the memory needed
for every block replica (when compressed oops is disabled) and in our new design the list
overhead will be per DatanodeStorageInfo and not per block replica.
> see attached design doc. for details and evaluation results.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message