hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15016) StoreServices facility in Region
Date Thu, 07 Jan 2016 20:47:39 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088075#comment-15088075

stack commented on HBASE-15016:

Thanks for taking the time to bring me along[~eshcar]

bq. Agree?

Agree on #1 and #2.

bq. Now, the region is responsible for decisions #3 and #4.

Thinking on it, being more specific (enabled by our helpful listing out of what is needed
in your comment above), it is not the Region that is responsible but the FlushPolicy implementation
that should be responsible. It is here where we should encapsulate all decision making about
flushing (If not possible, then there is something wrong with the FlushPolicy Interface and
we should fix it).

FlushPolicies are pluggable or we could just change the FlushLargeStorePolicy to accommodate
the compacting Store.

 * If all Stores in a Region are of the compacting type, just up the general region flush
threshold for the Table so the compacting Stores have more elbow room in which to do their
machinations, AND/OR
 * If a Region is a mix of Store types -- compacting and old-school non-compacting simple
Store types -- then the flush policy should give the Compacting Stores "...more memory space..."
and it should be more explicit that is doing this asking the Store its Type and acting on
this attribute directly rather than rely on a fuzzy 'active' memory used; e.g. flush a compacting
Store only if its 4 x this.flushSizeLowerBound AND we've already lushed all old-school Stores
> 1 x this.flushSizeLowerBound. You  have all the context you would need in the selectStoresToFlush
method I believe (you might want to expose more about Store internals but maybe you don't
need to if compacting Store is a configuration in HTableDescriptor or in HColumnDescriptor
-- these are available to you in the selectStoresToFlush context). If this would work for
you, then no need to change Region? Maybe I am missing something still.

Responding to your suggestion on how to bridge our approaches:

bq. ....A Region defines two thresholds and maintains a bit (flag) indicating whether or not
it has compacted memstores.
bq. Can we agree on this solution stack?

I put off agreeing to the above until there is finding that the above suggested FlushPolicy
approach won't work (FlushPolicy can ask if any of its Stores are of the compacting type on
initialization. This would be your "bit (flag)". Regards the two thresholds up in Region,
on review, this seems an indirect means of you obtaining your needs; be explicit about what
you are at inside in the FlushPolicy selection of Stores to compact.)

Thanks [~eshcar]

> StoreServices facility in Region
> --------------------------------
>                 Key: HBASE-15016
>                 URL: https://issues.apache.org/jira/browse/HBASE-15016
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>         Attachments: HBASE-15016-V01.patch, HBASE-15016-V02.patch, HBASE-15016-V03.patch,
Regioncounters.pdf, suggestion.patch
> The default implementation of a memstore ensures that between two flushes the memstore
size increases monotonically. Supporting new memstores that store data in different formats
(specifically, compressed), or that allows to eliminate data redundancies in memory (e.g.,
via compaction), means that the size of the data stored in memory can decrease even between
two flushes. This requires memstores to have access to facilities that manipulate region counters
and synchronization.
> This subtasks introduces a new region interface -- StoreServices, through which store
components can access these facilities.

This message was sent by Atlassian JIRA

View raw message