hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jinglun (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-14547) DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is not set.
Date Wed, 05 Jun 2019 04:06:00 GMT
Jinglun created HDFS-14547:

             Summary: DirectoryWithQuotaFeature.quota costs additional memory even the storage
type quota is not set.
                 Key: HDFS-14547
                 URL: https://issues.apache.org/jira/browse/HDFS-14547
             Project: Hadoop HDFS
          Issue Type: Improvement
    Affects Versions: 3.1.0
            Reporter: Jinglun

Our XiaoMi HDFS is considering upgrading from 2.6 to 3.1. We notice the storage type quota
'tsCounts' is instantiated to EnumCounters<StorageType>(StorageType.class), so it will
cost a long[5] even if we don't have any storage type quota on this inode(StorageType.class.getEnumConstants()
== 5).

In our cluster we have many dirs with quota and the NameNode's memory is in tension, so the
additional cost will be a problem.
See DirectoryWithQuotaFeature.Builder().

class DirectoryWithQuotaFeature$Builder {
  public Builder() {
   this.quota = new QuotaCounts.Builder().nameSpace(DEFAULT_NAMESPACE_QUOTA).
   typeSpaces(DEFAULT_STORAGE_SPACE_QUOTA).build();// set default value -1.
   this.usage = new QuotaCounts.Builder().nameSpace(1).build();
  public Builder typeSpaces(long val) {// set default value.
   return this;
class QuotaCounts$Builder {
  public Builder() {
    this.nsSsCounts = new EnumCounters<Quota>(Quota.class);
    this.tsCounts = new EnumCounters<StorageType>(StorageType.class);
class EnumCounters {
  public EnumCounters(final Class<E> enumClass) {
    final E[] enumConstants = enumClass.getEnumConstants();
    this.enumClass = enumClass;
    this.counters = new long[enumConstants.length];// new a long array here.
Related to HDFS-14542.


This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message