hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-5620) NameNode: implement Global ACL Set as a memory optimization.
Date Fri, 24 Jan 2014 22:13:39 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth resolved HDFS-5620.

    Resolution: Won't Fix

I'm resolving this issue as won't fix.  [~wheat9] was right to question whether or not we
really need this optimization.

Now that ACLs implementation is almost done, I can get a more accurate measurement of the
memory impact using jmap.  I'm including a plot that estimates memory consumption by ACLs
in a presumed 50 GB NameNode, trying to store up to 100 million inodes.  I plotted 3 different
usage scenarios:
# max usage: All inodes have an ACL, and each one has the maximum ACL entries.
# mid usage: Half the inodes have an ACL, and each ACL has 10 entries.
# low usage: 10% of inodes have an ACL, and each ACL has 2 entries.

For each scenario, I plotted the unoptimized memory consumption and also the optimized memory
consumption, assuming that only 20% of the ACLs are unique.  Each scenario uses a different
line color.  The unoptimized version uses a solid line, and the optimized version uses a dashed
line.  I'm also attaching the GnuPlot script I wrote to generate this.

The plot demonstrates that ACL usage really needs to get quite high (very large number of
inodes with a very high proportion of them having ACLs) before this memory optimization starts
to provide a benefit.

I am +1 for skipping this for now.  We can always resurrect this patch if we observe a need
for it based on real world usage of ACLs.  I'm also going to put this information into a new
revision of the design doc on HDFS-4685.

BTW, if we do resurrect this patch, then I probably wouldn't commit the exact HDFS-5620.1.patch
that I put together quickly as a prototype.  The Guava interner has a spinlock inside of it,
which we don't really need, because this would only be accessed under the namesystem write
lock anyway.

> NameNode: implement Global ACL Set as a memory optimization.
> ------------------------------------------------------------
>                 Key: HDFS-5620
>                 URL: https://issues.apache.org/jira/browse/HDFS-5620
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: HDFS ACLs (HDFS-4685)
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-5620.1.patch, aclRamSizingEstimates, aclRamSizingEstimates.png
> The {{AclManager}} can maintain a Global ACL Set to store all distinct ACLs in use by
the file system.  All inodes that have the same ACL entries can share the same ACL instance.

This message was sent by Atlassian JIRA

View raw message