hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-5620) NameNode: implement Global ACL Set as a memory optimization.
Date Fri, 24 Jan 2014 22:13:39 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Nauroth resolved HDFS-5620.
---------------------------------

    Resolution: Won't Fix

I'm resolving this issue as won't fix.  [~wheat9] was right to question whether or not we
really need this optimization.

Now that ACLs implementation is almost done, I can get a more accurate measurement of the
memory impact using jmap.  I'm including a plot that estimates memory consumption by ACLs
in a presumed 50 GB NameNode, trying to store up to 100 million inodes.  I plotted 3 different
usage scenarios:
# max usage: All inodes have an ACL, and each one has the maximum ACL entries.
# mid usage: Half the inodes have an ACL, and each ACL has 10 entries.
# low usage: 10% of inodes have an ACL, and each ACL has 2 entries.

For each scenario, I plotted the unoptimized memory consumption and also the optimized memory
consumption, assuming that only 20% of the ACLs are unique.  Each scenario uses a different
line color.  The unoptimized version uses a solid line, and the optimized version uses a dashed
line.  I'm also attaching the GnuPlot script I wrote to generate this.

The plot demonstrates that ACL usage really needs to get quite high (very large number of
inodes with a very high proportion of them having ACLs) before this memory optimization starts
to provide a benefit.

I am +1 for skipping this for now.  We can always resurrect this patch if we observe a need
for it based on real world usage of ACLs.  I'm also going to put this information into a new
revision of the design doc on HDFS-4685.

BTW, if we do resurrect this patch, then I probably wouldn't commit the exact HDFS-5620.1.patch
that I put together quickly as a prototype.  The Guava interner has a spinlock inside of it,
which we don't really need, because this would only be accessed under the namesystem write
lock anyway.

> NameNode: implement Global ACL Set as a memory optimization.
> ------------------------------------------------------------
>
>                 Key: HDFS-5620
>                 URL: https://issues.apache.org/jira/browse/HDFS-5620
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: HDFS ACLs (HDFS-4685)
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-5620.1.patch, aclRamSizingEstimates, aclRamSizingEstimates.png
>
>
> The {{AclManager}} can maintain a Global ACL Set to store all distinct ACLs in use by
the file system.  All inodes that have the same ACL entries can share the same ACL instance.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message