hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-7843) enable encapsulating compaction policy/compactor/store file manager interaction shennanigans
Date Thu, 21 Feb 2013 23:12:15 GMT

     [ https://issues.apache.org/jira/browse/HBASE-7843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Sergey Shelukhin updated HBASE-7843:

    Attachment: HBASE-7843-v0.patch

I will submit /r/; tests are running (many passed :)).
The idea of the patch is as follows.
Right now, we have CompactionRequest, which combines some logical properties of compaction,
some physical details necessary to execute it (store, region), and execution details pertaining
to CompactSplitThread. We also have CompactSelection which has a subset of logical properties
of CompactionRequest; until recently it used to manage off-peak compactions, but that was
moved, so it has almost no logic. Also, CompactionRequest can, since recently, be subclassed
by coprocessors.

The main purpose of this patch is to encapsulate the flow of compaction - from getting files
to select from thru selection and compaction.
To that end, and for general cleanup, the following was done.
1) CompactSelection is gone; its few fields are merged into CompactionRequest.
2) HStore, HRegionServer, and HRegion are gone from CompactionRequest via variety of means,
mostly (3).
3) CompactionSplitThread-related logic of CompactionRequest was moved into CompactionSplitThread's
nested class "CompactionRunner".
4) StoreEngine now creates CompactionContext class (could just be called "Compaction"), which
has the request, as well as hidden details of policy/compactor/etc. that are used to achieve
the main purpose of the patch. This class replaces CompactionRequest in the calls to HRegion
and Store compact(...) methods. I initially wanted to make compaction context the orchestration
class for HRegion/HStore/etc., with protected abstract method for hidden details, but it seems
awkward now, so most of the logic will stay in HRegion/HStore, who will call CompactionContext::compact
instead of Compactor::compact where appropriate.
4a) For (4) (CompactionContext creation), StoreEngine was made generic on policy/compactor/sfm
types, to allow for protected strongly-typed fields instead of hidden generic type fields,
and to thus avoid casts.
4b) Also due to (4) some methods on CompactionPolicy class are no longer general and necessary.
5) Fixed issue in HStore::completeCompaction - it works now because it's always called with
one file, but it won't actually work with multiple files. That is because it's called with
single file multiple times by HRegion::compact, and yet it performs both single-file-replaced
logic (move into place), and multi-file logic (remove files compacting, add new files, etc.).

> enable encapsulating compaction policy/compactor/store file manager interaction shennanigans
> --------------------------------------------------------------------------------------------
>                 Key: HBASE-7843
>                 URL: https://issues.apache.org/jira/browse/HBASE-7843
>             Project: HBase
>          Issue Type: Improvement
>          Components: Compaction
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>            Priority: Critical
>         Attachments: HBASE-7843-v0.patch
> To avoid massive casting and/or deciphering of structures traveling between SFM, compaction
policy, and compactor in non-trivial compaction schemes like stripe or level, we need to make
interaction between themselves hidden.
> Elsewhere, the changes are being made to coprocessor for compactions that will make CompactionRequest
a limited-visibility class for users, with coprocessors being able to subclass and return
it. -This seems like a viable solution for the problem at hand too. Policy will (optionally)
subclass compaction request and return it. Instead of calling something.compact(req), req.compact()
will be called (with "something" already stored inside req as of now), after which, magic
will happen.-
> After merging that code I actually see that subclassing compactionrequest in both will
break the policy, or require bunch of ugly code in coprocessors (a subclass for every policy,
and telling them apart.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message