hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chia-Ping Tsai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18158) Two running in-memory compaction threads may lose data
Date Mon, 05 Jun 2017 11:22:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16036823#comment-16036823
] 

Chia-Ping Tsai commented on HBASE-18158:
----------------------------------------

bq. So I still do not see how NPE can happen. Would be happy for more clarifications...
The first thread will set the versionedList to null and set isInterrupted to false after completing
the compaction.
{code}
  private void releaseResources() {
    isInterrupted.set(false);
    versionedList = null;
  }
{code}
So the later thread may keep executing the MERGE policy.
{code}
      if (!isInterrupted.get()) {
        if (resultSwapped = compactingMemStore.swapCompactedSegments(
            versionedList, result, (action==Action.MERGE))) {
          // update the wal so it can be truncated and not get too long
          compactingMemStore.updateLowestUnflushedSequenceIdInWAL(true); // only if greater
        }
      }
{code}
This issue is due to the lower threshold of flush size so i think it won't happen in production.
I just hope to make TestAcid* stable.


> Two running in-memory compaction threads may lose data
> ------------------------------------------------------
>
>                 Key: HBASE-18158
>                 URL: https://issues.apache.org/jira/browse/HBASE-18158
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>            Reporter: Chia-Ping Tsai
>            Assignee: Chia-Ping Tsai
>             Fix For: 2.0.0
>
>         Attachments: HBASE-18158.v0.patch
>
>
> {code:title=CompactingMemStore.java|borderStyle=solid}
>   private void stopCompaction() {
>     if (inMemoryFlushInProgress.get()) {
>       compactor.stop();
>       inMemoryFlushInProgress.set(false);
>     }
>   }
> {code}
> The stopCompaction() set inMemoryFlushInProgress to false so there may be two in-memory
compaction threads which execute simultaneously. If there are two running InMemoryFlushRunnable,
the later InMemoryFlushRunnable may change the versionedList. 
> {code:title=MemStoreCompactor.java|borderStyle=solid}
>   public boolean start() throws IOException {
>     if (!compactingMemStore.hasImmutableSegments()) { // no compaction on empty pipeline
>       return false;
>     }
>     // get a snapshot of the list of the segments from the pipeline,
>     // this local copy of the list is marked with specific version
>     versionedList = compactingMemStore.getImmutableSegments();
> {code}
> And the first InMemoryFlushRunnable will use the chagned versionedList to remove the
corresponding segments.
> {code:title=MemStoreCompactor#doCompaction|borderStyle=solid}
>       if (!isInterrupted.get()) {
>         if (resultSwapped = compactingMemStore.swapCompactedSegments(
>             versionedList, result, (action==Action.MERGE))) {
>           // update the wal so it can be truncated and not get too long
>           compactingMemStore.updateLowestUnflushedSequenceIdInWAL(true); // only if greater
>         }
>       }
> {code}
> In conclusion, first InMemoryFlushRunnable will remove the worng segment. And the later
InMemoryFlushRunnable may introduce NPE because first InMemoryFlushRunnable set versionedList
to null after compaction.
> {code}
> Exception in thread "RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=45712-inmemoryCompactions-1496563908038"
java.lang.NullPointerException
>         at org.apache.hadoop.hbase.regionserver.CompactionPipeline.swap(CompactionPipeline.java:119)
>         at org.apache.hadoop.hbase.regionserver.CompactingMemStore.swapCompactedSegments(CompactingMemStore.java:283)
>         at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompaction(MemStoreCompactor.java:212)
>         at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.start(MemStoreCompactor.java:122)
>         at org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory(CompactingMemStore.java:388)
>         at org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run(CompactingMemStore.java:500)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message