hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2006) ability to support storing extended attributes per file
Date Fri, 27 May 2011 07:00:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040092#comment-13040092
] 

Aaron T. Myers commented on HDFS-2006:
--------------------------------------

One option to not waste memory resources for files/directories which do not have extended
attributes set on them would be to go the same route as the {{INodeDirectory}} vs. {{INodeDirectoryWithQuota}}
classes, i.e. have {{INodeFile}} and {{InodeFileWithXAttrs}}. Of course, this strategy requires
that we implement classes for the matrix of all the different possible combinations of extra
metadata types.

A reasonable question to ask is whether we should concern ourselves all that much with the
memory consumption of this, since the extended attributes will presumably be per-file. Since
the NN's memory is dedicated to relatively few files and relatively many blocks, it may not
matter much if we go with a fairly naive approach.

Dhruba, do you have any data as to what percentage of the NN heap in a typical cluster goes
to per-file metadata versus per-block metadata?

> ability to support storing extended attributes per file
> -------------------------------------------------------
>
>                 Key: HDFS-2006
>                 URL: https://issues.apache.org/jira/browse/HDFS-2006
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> It would be nice if HDFS provides a feature to store extended attributes for files, similar
to the one described here: http://en.wikipedia.org/wiki/Extended_file_attributes. 
> The challenge is that it has to be done in such a way that a site not using this feature
does not waste precious memory resources in the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message