hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amir Langer (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7244) Reduce Namenode memory using Flyweight pattern
Date Wed, 12 Nov 2014 07:15:34 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Amir Langer updated HDFS-7244:
    Attachment: SampleCode.tar.gz

attached doc that describes an outline of the idea + Some sample code of how can we use the
slab to store BlockInfo objects.

> Reduce Namenode memory using Flyweight pattern
> ----------------------------------------------
>                 Key: HDFS-7244
>                 URL: https://issues.apache.org/jira/browse/HDFS-7244
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Amir Langer
>         Attachments: HDFS-7244.docx, SampleCode.tar.gz
> Using the flyweight pattern can dramatically reduce memory usage in the Namenode. The
pattern also abstracts the actual storage type and allows the decision of whether it is off-heap
or not and what is the serialisation mechanism to be configured per deployment. 
> The idea is to move all BlockInfo data (as a first step) to this storage using the Flyweight
pattern. The cost to doing it will be in higher latency when accessing/modifying a block.
The idea is that this will be offset with a reduction in memory and in the case of off-heap,
a dramatic reduction in memory (effectively, memory used for BlockInfo would reduce to a very
small constant value).
> This reduction will also have an huge impact on the latency as GC pauses will be reduced
considerably and may even end up with better latency results than the original code.
> I wrote a stand-alone project as a proof of concept, to show the pattern, the data structure
we can use and what will be the performance costs of this approach.
> see [Slab|https://github.com/langera/slab]
> and [Slab performance results|https://github.com/langera/slab/wiki/Performance-Results].
> Slab abstracts the storage, gives several storage implementations and implements the
flyweight pattern for the application (Namenode in our case).
> The stages to incorporate Slab into the Namenode is outlined in the sub-tasks JIRAs.

This message was sent by Atlassian JIRA

View raw message