hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
Date Mon, 24 Aug 2015 13:41:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709301#comment-14709301
] 

Yi Liu commented on HDFS-8900:
------------------------------

{quote}
The max size of the packed byte[] is 32KB * 32 = 1MB...
But such big sequential memory allocation could cause full GC when heap is fragmented.
{quote}
We can say this would never happen in real case. Typical the xattr value size is very small
(several bytes to tens of bytes, much smaller than 32KB, this is also the design goal of xattr,
for system/security namespace xattr used by HDFS core, the size are very small today; if applications
use user/trusted xattr to store large size value, the memory of NN itself will have serious
problem, for 1MB size xattrs of an INode as in your example, 1TB memory is required if there
is only 1MB number Inodes). In real case, even for HDFS transparent encryption which uses
xattr to store lots of information (see FileEncryptionInfo), the xattr value size is only
48bytes.

Of course the applications may use the xattr wrongly and use it to store large size (several
KB to tens of KB) value, to prevent this, I will add a threshold not to pack those large size
xattrs and still store them separately in a list as original.  For small size xattr, we will
pack them.  The threshold can be 1KB, in a common situation, the list storing large size xattr
is null, so only 4 bytes. 


> Compact XAttrs to optimize memory footprint.
> --------------------------------------------
>
>                 Key: HDFS-8900
>                 URL: https://issues.apache.org/jira/browse/HDFS-8900
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>         Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch, HDFS-8900.003.patch
>
>
> {code}
> private final ImmutableList<XAttr> xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point of view,
since {{ImmutableList}} and {{XAttr}} have object memory overhead, and each object has memory
alignment. 
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message