hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jinglun (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-14547) DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is not set.
Date Wed, 05 Jun 2019 04:06:00 GMT
Jinglun created HDFS-14547:
------------------------------

             Summary: DirectoryWithQuotaFeature.quota costs additional memory even the storage
type quota is not set.
                 Key: HDFS-14547
                 URL: https://issues.apache.org/jira/browse/HDFS-14547
             Project: Hadoop HDFS
          Issue Type: Improvement
    Affects Versions: 3.1.0
            Reporter: Jinglun


Our XiaoMi HDFS is considering upgrading from 2.6 to 3.1. We notice the storage type quota
'tsCounts' is instantiated to EnumCounters<StorageType>(StorageType.class), so it will
cost a long[5] even if we don't have any storage type quota on this inode(StorageType.class.getEnumConstants()
== 5).

In our cluster we have many dirs with quota and the NameNode's memory is in tension, so the
additional cost will be a problem.
See DirectoryWithQuotaFeature.Builder().

 
{code:java}
class DirectoryWithQuotaFeature$Builder {
  public Builder() {
   this.quota = new QuotaCounts.Builder().nameSpace(DEFAULT_NAMESPACE_QUOTA).
   storageSpace(DEFAULT_STORAGE_SPACE_QUOTA).
   typeSpaces(DEFAULT_STORAGE_SPACE_QUOTA).build();// set default value -1.
   this.usage = new QuotaCounts.Builder().nameSpace(1).build();
  }
  public Builder typeSpaces(long val) {// set default value.
   this.tsCounts.reset(val);
   return this;
  }
}
class QuotaCounts$Builder {
  public Builder() {
    this.nsSsCounts = new EnumCounters<Quota>(Quota.class);
    this.tsCounts = new EnumCounters<StorageType>(StorageType.class);
  }
}
class EnumCounters {
  public EnumCounters(final Class<E> enumClass) {
    final E[] enumConstants = enumClass.getEnumConstants();
    Preconditions.checkNotNull(enumConstants);
    this.enumClass = enumClass;
    this.counters = new long[enumConstants.length];// new a long array here.
  }
}
{code}
Related to HDFS-14542.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org


Mime
View raw message