hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16961) FileSystem Quotas
Date Fri, 28 Apr 2017 12:26:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988733#comment-15988733

Josh Elser commented on HBASE-16961:

Sorry, for the delayed reply and thanks for taking a look [~apurtell].

bq. 1. About the optional MasterObserver for automatically deleting quotas when the table
is deleted. Why do we not want this in general? Make it a part of core master, optionally

Mostly, this came in later; I think that's why I optionally enabled it. I don't have a good
reason that it should be disable by default.

bq. 2. Under what circumstances is NO_WRITES_COMPACTIONS advisable instead of NO_WRITES? 

It comes down to how strict administrators want to be with disk usage. As we move towards
a reality where space use is closely tracked, we might also want to avoid temporary spikes
in space usage for the compaction's file set (e.g. a compaction of three files, we start out
with space use of those three files, but then create a new file which would be similar in
size. So, we'd actually have ~2x the space taken, until the original 3 files are removed).

bq. 3. How would quotas and system recovery actions interact? Are quota checks bypassed for
actions taken by a superuser?

I actually haven't spent much time investigating this. I'm not sure if I've added short-circuits
for hbase.superusers. Did you have some specific commands in mind WRT "system recovery actions"?
I could probably whip up a unit test without much hassle.

bq. Yes, this could be a problem, if we are trying to apply quotas to more than one table
as part of activity to bring something runaway under control. Not urgent, but let's make sure
there's a follow up JIRA for this. 

Will create and link here.

> FileSystem Quotas
> -----------------
>                 Key: HBASE-16961
>                 URL: https://issues.apache.org/jira/browse/HBASE-16961
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>         Attachments: hbase-quota-test.sh
> Umbrella issue for tracking the filesystem utilization of HBase data, defining quotas
on that utilization, and enforcement when utilization exceeds the limits of the quota.
> At a high level: we can define quotas on tables and namespaces. Region size is computed
by RegionServers and sent to the Master. The Master inspects the sizes of Regions, rolling
up to table and namespace sizes. Defined quotas in the quota table are evaluated given the
computed sizes, and, for those tables/namespaces violating the quota, RegionServers are informed
to take some action to limit any further filesystem growth by that table/namespace.
> Discuss: https://lists.apache.org/thread.html/66a4b0c3725b5cbdd61dd6111c43847adaeef7b7da5f4cd045df30ef@%3Cdev.hbase.apache.org%3E
> Design Doc: http://home.apache.org/~elserj/hbase/FileSystemQuotasforApacheHBase.pdf or

This message was sent by Atlassian JIRA

View raw message