hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8900) Optimize XAttr memory footprint.
Date Thu, 20 Aug 2015 21:41:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705813#comment-14705813

Andrew Wang commented on HDFS-8900:

I talked with [~cmccabe] about compatibility offline, we had a few thoughts:

* Unlimited size xattrs are an anti-pattern and not that useful. Same for very large xattrs
in general.
* For branch-2, one compatible approach is to size the length field based on the max size
config, up to a full 4-byte signed integer. This is what "unlimited" currently means anyway.
Then, do a trunk follow-on to just use a 2-byte length and maybe get rid of the max size config
in favor of the hard limit.
* Practically speaking, it's unlikely anyone's ever changed the max size, so the impact of
the incompatibility is limited. Patch might be fine for branch-2 as is.

> Optimize XAttr memory footprint.
> --------------------------------
>                 Key: HDFS-8900
>                 URL: https://issues.apache.org/jira/browse/HDFS-8900
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>         Attachments: HDFS-8900.001.patch
> {code}
> private final ImmutableList<XAttr> xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point of view,
since {{ImmutableList}} and {{XAttr}} have object memory overhead, and each object has memory
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.

This message was sent by Atlassian JIRA

View raw message