hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16961) FileSystem Quotas
Date Fri, 28 Apr 2017 20:04:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989385#comment-15989385

Josh Elser commented on HBASE-16961:

I was chatting with Clay B. today offline, and he had suggested that showing some of the new
shell commands and output would also help. I also struggle a little bit, differentiating between
kind of shell commands are good for 1) what I want to know as the developers, 2) what someone
verifying the feature wants to know and 3) what a real sysadmin would want to know. Maybe
this will elicit some comments :)

hbase(main):002:0> help 'list_quota_snapshots'
Lists the current snapshot of quotas on the given RegionServer. This
information filters to each RegionServer from the Master. For each
table, a snapshot includes the filesystem use, the filesystem limit,
and the policy to enact when the limit is exceeded. This command is
useful for debugging the running state of a cluster using filesystem quotas.

For example:

    hbase> list_quota_snapshots 'regionserver1.domain,16020,1483482894742'

hbase(main):004:0> list_quota_snapshots 'hw10447.local,16201'
 joshns:quota3 3287236897 3221225472 true NO_WRITES
 joshns:quota4 3287236897 3221225472 true NO_WRITES
 t1 3570208 5368709120 false None
 joshns:quota1 3287236897 3221225472 true NO_WRITES
 joshns:quota2 3287236897 3221225472 true NO_WRITES
5 row(s)
Took 0.3900 seconds

hbase(main):003:0> help 'list_quota_table_sizes'
Lists the sizes of the tables in HBase as collected
for the purpose of implementing filesystem utilization
quotas. This information is extracted from the HBase
Master and drives future quota actions in the cluster.

For example:

    hbase> list_quota_table_sizes

hbase(main):005:0> list_quota_table_sizes
TABLE                                         SIZE
 hbase:meta                                   311497
 hbase:namespace                              4958
 usertable                                    5640726867
 joshns:quota3                                1029790723
 t1                                           2411100
 joshns:quota4                                194999608
 joshns:quota1                                1031962976
 joshns:quota2                                1030483590
 hbase:quota                                  20079
9 row(s)
Took 0.0630 seconds

hbase(main):004:0> help 'list_quota_violations'
Lists the current quota violations being enforced by a RegionServer.
Violations are enacted based on the quota snapshot information a RegionServer
holds (see list_quota_snapshots). Each violation contains the action the
RegionServer is taking on the table. This command is useful in debugging
the running state of a cluster using filesystem quotas.

For example:

    hbase> list_quota_violations 'regionserver1.domain,16020,1483482894742'

hbase(main):006:0> list_quota_violations 'hw10447.local,16201'
TABLE                                         POLICY
 joshns:quota3                                NO_WRITES
 joshns:quota4                                NO_WRITES
 joshns:quota1                                NO_WRITES
 joshns:quota2                                NO_WRITES
4 row(s)
Took 0.0190 seconds

> FileSystem Quotas
> -----------------
>                 Key: HBASE-16961
>                 URL: https://issues.apache.org/jira/browse/HBASE-16961
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>         Attachments: hbase-quota-test.sh
> Umbrella issue for tracking the filesystem utilization of HBase data, defining quotas
on that utilization, and enforcement when utilization exceeds the limits of the quota.
> At a high level: we can define quotas on tables and namespaces. Region size is computed
by RegionServers and sent to the Master. The Master inspects the sizes of Regions, rolling
up to table and namespace sizes. Defined quotas in the quota table are evaluated given the
computed sizes, and, for those tables/namespaces violating the quota, RegionServers are informed
to take some action to limit any further filesystem growth by that table/namespace.
> Discuss: https://lists.apache.org/thread.html/66a4b0c3725b5cbdd61dd6111c43847adaeef7b7da5f4cd045df30ef@%3Cdev.hbase.apache.org%3E
> Design Doc: http://home.apache.org/~elserj/hbase/FileSystemQuotasforApacheHBase.pdf or

This message was sent by Atlassian JIRA

View raw message