hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14920) Compacting Memstore
Date Tue, 29 Mar 2016 19:40:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216720#comment-15216720

stack commented on HBASE-14920:

I like the notion of a 'CompactionMemStore' instead of a plain old MemStore.

Reading the nice description made me think of how we need to work on recovery. There'll be
more to recover if we can carry more in memory.  Another issue.

This needs a comment on it: IN_MEMORY_FLUSH_THRESHOLD_FACTOR

Would be good if could use the Interface Store rather than implementation HStore in below

66	  private HStore store;

For these...

	  // A flag for tests only

... we normally just flag with @VisibleForTesting annotation. FYI.

I've asked this before but forgot the answer, this has to be here: isCompactingMemStore ?

Is this a race in the below?

184	  @Override public long getFlushableSize() {
185	    long snapshotSize = getSnapshot().getSize();
186	    if(snapshotSize == 0) {
187	      //if snapshot is empty the tail of the pipeline is flushed
188	      snapshotSize = pipeline.getTailSize();
189	    }
190	    return snapshotSize > 0 ? snapshotSize : keySize();
191	  }

If no snapshot, use the tail of the pipeline? The pipeline tail has not yet been moved to
be the new snapshot?

If sequenceid is MAX, don't we want to update it?

	    if(minSequenceId != Long.MAX_VALUE) {

I asked this before too...so, to do compaction, we need to have exclusive lock or exclusive
update lock just while you swap segments (the compacted for the inputs?)... it looks like
the next method, flushInMemory, answers my question so ignore..

245	      /* The thread is dispatched to flush-in-memory. This cannot be done
246	      * on the same thread, because for flush-in-memory we require updatesLock
247	      * in exclusive mode while this method (checkActiveSize) is invoked holding updatesLock
248	      * in the shared mode. */

Any guard against queuing same old compactions over and over again?

	250	      if(pool != null) { // the pool can be null in some tests scenarios
251	        InMemoryFlushWorker worker = new InMemoryFlushWorker();
252	        LOG.info("Dispatching the MemStore in-memory flush for store "
253	            + store.getColumnFamilyName());
254	        pool.submit(worker);
255	        inMemoryFlushInProgress.set(true);
256	      }

... say there is a bug or we are slow to run already-queued compactions?


The below would seem to say we compact with updates lock held. Fix if wrong... if it is true,
then lets discuss:

351	  * It takes the updatesLock exclusively, pushes active into the pipeline,
352	  * and compacts the pipeline.

Got as far as MemstoreCompactor... will be back.

> Compacting Memstore
> -------------------
>                 Key: HBASE-14920
>                 URL: https://issues.apache.org/jira/browse/HBASE-14920
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>         Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch
> Implementation of a new compacting memstore with non-optimized immutable segment representation

This message was sent by Atlassian JIRA

View raw message