hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yiqun Lin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12934) RBF: Federation supports global quota
Date Mon, 01 Jan 2018 12:50:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16307398#comment-16307398
] 

Yiqun Lin edited comment on HDFS-12934 at 1/1/18 12:49 PM:
-----------------------------------------------------------

Hi [~elgoiri], following are my comments:

bq. I've checked MountTableResolver and it might make sense to use a similar TreeMap for the
map in RouterQuotaLocalCache. It wo...
Using TreeMap will benefit for function {{getChildrenPath}} I think. I don't think this can
simplify the recursive search for function {{RouterQuotaLocalCache#getQuotaUsage}}. Am I missing
something? In additionally, use the read/write lock to make this thread-safe to use.

bq. In Router, I would rename getChildrenPaths(String path) into getQuotaUsage(String path)
and getQuotaUsageCache() getQuotaUsage()..
I didn't fully understanding on this. Do you mean following changes:
{noformat}
getChildrenPaths -->getQuotaUsage
getQuotaUsageCache-->getQuotaUsage
{noformat}
The first change looks strange. I only renamed getQuotaUsageCache to getQuotaUsage.

{quote}
Typo NamaService in RouterRpcServer and loc.getNameserviceId() should be extracted.
Update TestMountTable to include the new field for the quota.
{quote}
Addressed.

{quote}
In TestRouterQuota, I think we shouldn't use the wait for breaking the quota but actually
create successfully the first 3 and then fail in the next one.
{quote}
Since quota cache service is a periodical service, we should ensure the mount tables are loaded
and their quota are updated. So I use {{GenericTestUtils#waitFor}} for waiting this instead
of just sleeping fixed time. Another way, we don't need to wait for quota being updated, we
can just call {{periodicInvoke}} for trigger this. But this is not a simulated real scene
that we want to test.
I found test of setQuota/getQuota can be simplified. Waiting operation is not needed. I have
updated on this.

bq. Do you mind opening the JIRA for the doc and the UI?
Had filed JIRA HDFS-12972 and HDFS-12973 for tracking this.

Attach the updated patch.




was (Author: linyiqun):
Hi [~elgoiri], following are my comments:

bq. I've checked MountTableResolver and it might make sense to use a similar TreeMap for the
map in RouterQuotaLocalCache. It wo...
Using TreeMap will benefit for function {{getChildrenPath}} I think. I don't think this can
simplify the recursive search for function {{RouterQuotaLocalCache#getQuotaUsage}}. Am I missing
something? In additionally, use the read/write lock to make this thread-safe to use.

bq. In Router, I would rename getChildrenPaths(String path) into getQuotaUsage(String path)
and getQuotaUsageCache() getQuotaUsage()..
I didn't fully understanding on this. Do you mean following changes:
{noformat}
getChildrenPaths -->getQuotaUsage
getQuotaUsageCache-->getQuotaUsage
{noformat}
The first change looks strange. I only renamed getQuotaUsageCache to getQuotaUsage.

{quote}
Typo NamaService in RouterRpcServer and loc.getNameserviceId() should be extracted.
Update TestMountTable to include the new field for the quota.
{quote}
Addressed.

{quote}
In TestRouterQuota, I think we shouldn't use the wait for breaking the quota but actually
create successfully the first 3 and then fail in the next one.
{quote}
Since quota cache service is a periodical service, we should ensure the mount tables are loaded
and their quota are updated. So I use {{GenericTestUtils#waitFor}} for waiting this instead
of just sleeping fixed time. This is a simulated real scene. I found test of setQuota/getQuota
can be simplified. Waiting operation is not needed. I have updated on this.

bq. Do you mind opening the JIRA for the doc and the UI?
Had filed JIRA HDFS-12972 and HDFS-12973 for tracking this.

Attach the updated patch.



> RBF: Federation supports global quota
> -------------------------------------
>
>                 Key: HDFS-12934
>                 URL: https://issues.apache.org/jira/browse/HDFS-12934
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: 3.0.0
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>              Labels: RBF
>         Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch, HDFS-12934.003.patch,
HDFS-12934.004.patch, RBF support  global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder. Currently the quota
will be applied for each subcluster under the specified folder via RPC call.
> It will be very useful for users that federation can support setting global quota and
exposing the command of this.
> In a federated environment, a folder can be spread across multiple subclusters. For this
reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can exceed
maximun quota value.
> # We need to construct one <Path, QuotaUsage> cache map for storing the sum  quota
usage of these subclusters under federation folder. Every time we want to do WRITE operation
under specified folder, we will get its quota usage from cache and verify its quota. If quota
exceeded, throw exception, otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The set/unset
command will be like:
> {noformat}
>  hdfs dfsrouteradmin -setQuota -ns <nsQuota> -ss <ssQuota> <mount table>
>  hdfs dfsrouteradmin -clrQuota  <mount table>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message