hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jinglun (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-14547) DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is not set.
Date Fri, 14 Jun 2019 03:03:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-14547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jinglun updated HDFS-14547:
---------------------------
    Attachment:     (was: HDFS-14547-Test Report.pdf)

> DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is
not set.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HDFS-14547
>                 URL: https://issues.apache.org/jira/browse/HDFS-14547
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 3.1.0
>            Reporter: Jinglun
>            Assignee: Jinglun
>            Priority: Major
>         Attachments: HDFS-14547-design, HDFS-14547-patch003-Test Report.pdf, HDFS-14547.001.patch,
HDFS-14547.002.patch
>
>
> Our XiaoMi HDFS is considering upgrading from 2.6 to 3.1. We notice the storage type
quota 'tsCounts' is instantiated to EnumCounters<StorageType>(StorageType.class), so
it will cost a long[5] even if we don't have any storage type quota on this inode(only space
quota or name quota).
> In our cluster we have many dirs with quota and the NameNode's memory is in tension,
so the additional cost will be a problem.
>  See DirectoryWithQuotaFeature.Builder().
>  
> {code:java}
> class DirectoryWithQuotaFeature$Builder {
>   public Builder() {
>    this.quota = new QuotaCounts.Builder().nameSpace(DEFAULT_NAMESPACE_QUOTA).
>    storageSpace(DEFAULT_STORAGE_SPACE_QUOTA).
>    typeSpaces(DEFAULT_STORAGE_SPACE_QUOTA).build();// set default value -1.
>    this.usage = new QuotaCounts.Builder().nameSpace(1).build();
>   }
>   public Builder typeSpaces(long val) {// set default value.
>    this.tsCounts.reset(val);
>    return this;
>   }
> }
> class QuotaCounts$Builder {
>   public Builder() {
>     this.nsSsCounts = new EnumCounters<Quota>(Quota.class);
>     this.tsCounts = new EnumCounters<StorageType>(StorageType.class);
>   }
> }
> class EnumCounters {
>   public EnumCounters(final Class<E> enumClass) {
>     final E[] enumConstants = enumClass.getEnumConstants();
>     Preconditions.checkNotNull(enumConstants);
>     this.enumClass = enumClass;
>     this.counters = new long[enumConstants.length];// new a long array here.
>   }
> }
> {code}
> Related to HDFS-14542.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message