hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anastasia Braginsky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16156) In write heavy scenario creating in memory flushes leads to contention
Date Sat, 02 Jul 2016 08:50:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360084#comment-15360084
] 

Anastasia Braginsky commented on HBASE-16156:
---------------------------------------------

Hi Guys,

I have just seen this and HBASE-16162. Both JIRAs have the same root. Indeed there is a bug
in synchronization of dispatching the *single* compaction thread. This includes that active
should be pushed to the compaction pipeline only once. We have seen this synch problem and
solved it like this:

private boolean shouldFlushInMemory() {
    if(getActive().getSize() > inmemoryFlushSize) { // size above flush threshold
      if (allowCompaction.get())      // the compaction is allowed in the test case
        // the inMemoryFlushInProgress is CASed to be true here in order to mutual exclude
        // the insert of the active into the compaction pipeline
        return (inMemoryFlushInProgress.compareAndSet(false,true)); // the winner is the only
thread who is going to deal with compaction
    }
    return false;
  }

protected void checkActiveSize() {
    if (shouldFlushInMemory()) {
      /* The thread is dispatched to flush-in-memory. This cannot be done
      * on the same thread, because for flush-in-memory we require updatesLock
      * in exclusive mode while this method (checkActiveSize) is invoked holding updatesLock
      * in the shared mode. */
      InMemoryFlushRunnable runnable = new InMemoryFlushRunnable();
      LOG.info("Dispatching the MemStore in-memory flush for store " + store.getColumnFamilyName());
      getPool().execute(runnable);
    }
  }

// internally used method, externally visible only for tests
  // when invoked directly from tests it must be verified that the caller doesn't hold updatesLock,
  // otherwise there is a deadlock
  @VisibleForTesting
  void flushInMemory() throws IOException {
    // Phase I: Update the pipeline
    getRegionServices().blockUpdates();
    try {
      MutableSegment active = getActive();
      LOG.info("IN-MEMORY FLUSH: Pushing active segment into compaction pipeline, " +
          "and initiating compaction.");
      pushActiveToPipeline(active);
    } finally {
      getRegionServices().unblockUpdates();
    }
    // Phase II: Compact the pipeline
    try {
      if (allowCompaction.get()) {
        // setting the inMemoryFlushInProgress flag again for the case this method is invoked
        // directly (only in tests) in the common path setting from true to true is idempotent
        inMemoryFlushInProgress.set(true);
        // Speculative compaction execution, may be interrupted if flush is forced while
        // compaction is in progress
        compactor.start();
      }
    } catch (IOException e) {
      LOG.warn("Unable to run memstore compaction. region "
          + getRegionServices().getRegionInfo().getRegionNameAsString()
          + "store: "+ getFamilyName(), e);
    } finally {
      stopCompaction();
    }
  }

After the fix we didn't see synch issues. This is included in HBASE-14921 patch that is going
to be released tomorrow. But truly we should have done it as separate fix...

> In write heavy scenario creating in memory flushes leads to contention
> ----------------------------------------------------------------------
>
>                 Key: HBASE-16156
>                 URL: https://issues.apache.org/jira/browse/HBASE-16156
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>             Fix For: 2.0.0
>
>
> In write heavy cases, the inmemory flushes blocks the write because it takes a update
lock. Dumps show that it is causing heavy contention and leads to memstore filling up and
so intern blocks the flush from happening sooner. This JIRA is to discuss optimal settings
and then make suitable changes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message