hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
Date Thu, 30 Oct 2014 23:13:34 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191008#comment-14191008
] 

stack commented on HBASE-10201:
-------------------------------

I wrote the list to get another opinion on the patch pre-commit.

Has this patch been deployed somewhere in production (smile?).  If so, would be good to know.
 In production, it helps?

On rereview:

" This value should be less than half of the total memstore
+    threshold (hbase.hregion.memstore.flush.size)."

Do we ensure this in code?  If not, should we?

bq.  I think it is better to open another issue to handle the duplication.

Can you do this for the accounting fixup so by-Store in HLog.

Should log when we do this:

+    long columnfamilyFlushSize = this.htableDescriptor
+        .getMemStoreColumnFamilyFlushSize();
+    if (columnfamilyFlushSize <= 0) {
+      columnfamilyFlushSize = conf.getLong(
+          HConstants.HREGION_MEMSTORE_COLUMNFAMILY_FLUSH_SIZE_LOWER_BOUND,
+          HTableDescriptor.DEFAULT_MEMSTORE_COLUMNFAMILY_FLUSH_SIZE_LOWER_BOUND);

I can add on commit unless we are doing a new version.

This does not have to be public since it is used from same package:

+  public long getEarliestFlushTimeForAllStores() {

ditto this

getLatestFlushTimeForAllStores

And this ... isPerColumnFamilyFlushEnabled

nit: Guard debug logging with an if LOG.isDebugEnabled... +            LOG.debug("Since none
of the CFs were above the size, flushing all.");

When we flush, we write the sequenceid flush to WAL.  This patch should have no effect on
it.

Sequenceids are region scoped.  If we flush by Store, will there be holes in our accounting?

For example, given 3 column families, A, B, and C.

I write sequenceid 1 to A, sequenceid 2 to B, and sequenceid 3 to C.  I then write sequence
4 to A.  The edit at sequenceid 4 is big and pushes us over and brings on a flush.  We flush
A and edits 1 and 4.  Is the fact that edits 2 and 3 are still up in memory going to mess
us up.... Say the server crashes, at replay time we see we flushed up to edit 4, will we think
that we edits 2 and 3 persisted? If you don't have an answer, I can work on the answer.



> Port 'Make flush decisions per column family' to trunk
> ------------------------------------------------------
>
>                 Key: HBASE-10201
>                 URL: https://issues.apache.org/jira/browse/HBASE-10201
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>            Reporter: Ted Yu
>            Assignee: zhangduo
>            Priority: Critical
>             Fix For: 2.0.0, 0.99.2
>
>         Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, HBASE-10201-0.98_1.patch,
HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, HBASE-10201.patch, HBASE-10201_1.patch,
HBASE-10201_2.patch, HBASE-10201_3.patch
>
>
> Currently the flush decision is made using the aggregate size of all column families.
When large and small column families co-exist, this causes many small flushes of the smaller
CF. We need to make per-CF flush decisions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message