hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names
Date Thu, 03 Jun 2010 20:27:56 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875282#action_12875282

Suresh Srinivas commented on HDFS-1110:

Thanks Jakob.

bq. The NameDictionary.lookup(name, value) method seems a bit odd in its usage. Both times
it's used via dictionary.lookup(name, name), which makes me wonder if this is the right API.
Do we expect NameDictionary  to be used elsewhere such that this abstraction is worth the
odd API?

I see your point. I decided to use same key and value to reduce object count. I do not want
to rule out using NameDictionary for other things and hence it is generic with the possibility
of key and value being different. I can add a comment to indicate key and value are the same
when doing lookup. Let me know if you think of other alternatives.

> Namenode heap optimization - reuse objects for commonly used file names
> -----------------------------------------------------------------------
>                 Key: HDFS-1110
>                 URL: https://issues.apache.org/jira/browse/HDFS-1110
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>         Attachments: hdfs-1110.2.patch, hdfs-1110.3.patch, hdfs-1110.patch
> There are a lot of common file names used in HDFS, mainly created by mapreduce, such
as file names starting with "part". Reusing byte[] corresponding to these recurring file names
will save significant heap space used for storing the file names in millions of INodeFile

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message