hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anastasia Braginsky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18158) Two running in-memory compaction threads may lose data
Date Mon, 05 Jun 2017 11:13:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16036815#comment-16036815
] 

Anastasia Braginsky commented on HBASE-18158:
---------------------------------------------

I understand that the concern is that the first compaction thread got to be interrupted, but
didn't really stopped yet and the second one succeed in CAS on inMemoryFlushInProgress and
is already starting. I also understand that two those threads have different VersionedList
of segments in pipeline, meaning also different VersionedList objects. If the first compaction
thread still succeeds to do the swap on the pipeline (compaction), the second one will not
do anything due to different version number. So I still do not see how NPE can happen. Would
be happy for more clarifications...

> Two running in-memory compaction threads may lose data
> ------------------------------------------------------
>
>                 Key: HBASE-18158
>                 URL: https://issues.apache.org/jira/browse/HBASE-18158
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>            Reporter: Chia-Ping Tsai
>            Assignee: Chia-Ping Tsai
>             Fix For: 2.0.0
>
>         Attachments: HBASE-18158.v0.patch
>
>
> {code:title=CompactingMemStore.java|borderStyle=solid}
>   private void stopCompaction() {
>     if (inMemoryFlushInProgress.get()) {
>       compactor.stop();
>       inMemoryFlushInProgress.set(false);
>     }
>   }
> {code}
> The stopCompaction() set inMemoryFlushInProgress to false so there may be two in-memory
compaction threads which execute simultaneously. If there are two running InMemoryFlushRunnable,
the later InMemoryFlushRunnable may change the versionedList. 
> {code:title=MemStoreCompactor.java|borderStyle=solid}
>   public boolean start() throws IOException {
>     if (!compactingMemStore.hasImmutableSegments()) { // no compaction on empty pipeline
>       return false;
>     }
>     // get a snapshot of the list of the segments from the pipeline,
>     // this local copy of the list is marked with specific version
>     versionedList = compactingMemStore.getImmutableSegments();
> {code}
> And the first InMemoryFlushRunnable will use the chagned versionedList to remove the
corresponding segments.
> {code:title=MemStoreCompactor#doCompaction|borderStyle=solid}
>       if (!isInterrupted.get()) {
>         if (resultSwapped = compactingMemStore.swapCompactedSegments(
>             versionedList, result, (action==Action.MERGE))) {
>           // update the wal so it can be truncated and not get too long
>           compactingMemStore.updateLowestUnflushedSequenceIdInWAL(true); // only if greater
>         }
>       }
> {code}
> In conclusion, first InMemoryFlushRunnable will remove the worng segment. And the later
InMemoryFlushRunnable may introduce NPE because first InMemoryFlushRunnable set versionedList
to null after compaction.
> {code}
> Exception in thread "RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=45712-inmemoryCompactions-1496563908038"
java.lang.NullPointerException
>         at org.apache.hadoop.hbase.regionserver.CompactionPipeline.swap(CompactionPipeline.java:119)
>         at org.apache.hadoop.hbase.regionserver.CompactingMemStore.swapCompactedSegments(CompactingMemStore.java:283)
>         at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompaction(MemStoreCompactor.java:212)
>         at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.start(MemStoreCompactor.java:122)
>         at org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory(CompactingMemStore.java:388)
>         at org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run(CompactingMemStore.java:500)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message