hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names
Date Thu, 10 Jun 2010 06:48:15 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877348#action_12877348

Konstantin Shvachko commented on HDFS-1110:

+1 on the patch.

We also talked about replacing HashMap with a TreeMap.
The advantages of TreeMap are 
- you don't need to wrap byte[] into a class, as it lets to provide a comparator, which compares
byte[]s, and
- it does not have memory overhead of HashMap

The disadvantage is that file creation will require a log-time lookup in TreeMap instead of
a constant lookup in HashMap. Besides, the HashMap memory overhead is small compared to the
overall memory savings provided by the approach. 
The decision is to use HashMap with a byte[] wrapped into ByteArray class.

> Namenode heap optimization - reuse objects for commonly used file names
> -----------------------------------------------------------------------
>                 Key: HDFS-1110
>                 URL: https://issues.apache.org/jira/browse/HDFS-1110
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>         Attachments: hdfs-1110.2.patch, hdfs-1110.3.patch, hdfs-1110.4.patch, hdfs-1110.5.patch,
hdfs-1110.6.patch, hdfs-1110.patch
> There are a lot of common file names used in HDFS, mainly created by mapreduce, such
as file names starting with "part". Reusing byte[] corresponding to these recurring file names
will save significant heap space used for storing the file names in millions of INodeFile

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message