hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yiqun Lin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-12934) RBF: Federation supports global quota
Date Mon, 18 Dec 2017 04:10:04 GMT

     [ https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yiqun Lin updated HDFS-12934:
-----------------------------
    Description: 
Now federation doesn't support set the global quota for each folder. Currently the quota will
be applied for each subcluster under the specified folder via RPC call.
It will be very useful for users that federation can support setting global quota and exposing
the command of this.

In a federated environment, a folder can be spread across multiple subclusters. For this reason,
we plan to solve this by following way:

# Set global quota across each subcluster. We don't allow each subcluster cab exceed maximun
quota value.
# We need to construct one <Path, QuotaUsage> cache map for storing the sum  quota usage
of these subclusters under federation folder. Every time we want to do WRITE operation under
specified folder, we will get its quota usage from cache and verify its quota. If quota exceeded,
throw exception, otherwise update its quota usage in cache when finishing operations.

The quota will be set to mount table and as a new field in mount table. The set/unset command
will be like:
{noformat}
 hdfs dfsrouteradmin -setQuota <nsQuota>  <ssQuota> <mount table>
 hdfs dfsrouteradmin -clrQuota  <mount table>
{noformat}

  was:
Now federation doesn't support set the global quota for each folder. Currently the quota will
be applied for each subcluster under the specified folder via RPC call.
It will be very useful for users that federation can support setting global quota and exposing
the command of this.

In a federated environment, a folder can be spread across multiple subclusters. For this reason,
we plan to solve this by following way:

# Set global quota across each subcluster. We don't allow each subcluster cab exceed maximun
quota value.
# We need to construct one <Path, QuotaUsage> cache map for storing current quota usage
of these subclusters, Every time we want to do WRITE operation under specified folder, we
will get its quota usage from cache and verify its quota. If quota exceeded, throw exception,
otherwise update its quota usage in cache when finishing operations.

The quota will be set to mount table and as a new field in mount table. The set/unset command
will be like:
{noformat}
 hdfs dfsrouteradmin -setQuota <nsQuota>  <ssQuota> <mount table>
 hdfs dfsrouteradmin -clrQuota  <mount table>
{noformat}


> RBF: Federation supports global quota
> -------------------------------------
>
>                 Key: HDFS-12934
>                 URL: https://issues.apache.org/jira/browse/HDFS-12934
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: 3.0.0
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>              Labels: RBF
>
> Now federation doesn't support set the global quota for each folder. Currently the quota
will be applied for each subcluster under the specified folder via RPC call.
> It will be very useful for users that federation can support setting global quota and
exposing the command of this.
> In a federated environment, a folder can be spread across multiple subclusters. For this
reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster cab exceed
maximun quota value.
> # We need to construct one <Path, QuotaUsage> cache map for storing the sum  quota
usage of these subclusters under federation folder. Every time we want to do WRITE operation
under specified folder, we will get its quota usage from cache and verify its quota. If quota
exceeded, throw exception, otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The set/unset
command will be like:
> {noformat}
>  hdfs dfsrouteradmin -setQuota <nsQuota>  <ssQuota> <mount table>
>  hdfs dfsrouteradmin -clrQuota  <mount table>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message