hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-20800) Master orchestrated compactions
Date Wed, 27 Jun 2018 21:27:00 GMT
stack created HBASE-20800:
-----------------------------

             Summary: Master orchestrated compactions
                 Key: HBASE-20800
                 URL: https://issues.apache.org/jira/browse/HBASE-20800
             Project: HBase
          Issue Type: Umbrella
          Components: Compaction
            Reporter: stack
            Assignee: Mohit Goel


An umbrella issue for having compactions go via the Master so we can have a centralized arbitrator
of cluster i/o. If we put Master in the way, we can do stuff like:

 * Ask the Master for current cluster compaction state; what is running, what is blocked
 * Master can manage cluster-wide compaction policy and/or throttling/or blocking of compaction
i/os.
 * Master can schedule when and where compactions run so we can guard against the pathological
where all RegionServers decide now is the time to major compact bringing on a compaction storm.

Other side-benefits might include being able to farm out the compaction work to another process
-- e.g. the splice machine model of having spark run the compactions -- or just to a separate
compactor that we might i/o nice.

 * We'll need to figure how to externalize the CompactionRequest so it can be passed over
RPC.
 * We'll need to have something like a CompactionManager in the Master process that keeps
up current cluster state.


MOB needs a compaction fabric it can use. Its compactions are currently Master-based only
and so don't scale. It could make use of this mechanism to ask the Master to farm out its
compaction requests.

This is an umbrella issue. I thought I'd filed one already on this topic but can't find it.
Will shut it down if I trip over it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message