hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eshcar Hillel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17379) Lack of synchronization in CompactionPipeline#getScanners()
Date Mon, 02 Jan 2017 10:38:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15792644#comment-15792644
] 

Eshcar Hillel commented on HBASE-17379:
---------------------------------------

Looking at the current trunk getTailSize() is only invoked from HStore::getSizeToFlush()->CompactingMemStore::getFlushableSize(),
and the logic is equivalent to what happens in DefaultMemStore.

The method getMinSequenceId() is invoked from updateLowestUnflushedSequenceIdInWAL().
The compacting pipeline maintains the following invariant ([~stack], [~Apache9], see if you
agree with me):
Consider the series of segments at the tail of a pipeline over time. That is, we look at a
specific pipeline over time and whenever its tail changes we add this segment tail to the
end of the series. 
If we look at the minimal sequence number in all these segments by their order in the series
we get a monotonically increasing series of numbers. 
Both flush and compaction maintain this invariant.
So when we get the minimal number of the tail, even with no lock, worst case scenario we retrieve
a number that is *smaller* than the current minimal sequence number and we leave in the WAL
some entries that could have been truncated. But this ok. They can be truncated next time
we update the WAL.


> Lack of synchronization in CompactionPipeline#getScanners()
> -----------------------------------------------------------
>
>                 Key: HBASE-17379
>                 URL: https://issues.apache.org/jira/browse/HBASE-17379
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>         Attachments: 17379.v1.txt, 17379.v14.txt, 17379.v2.txt, 17379.v3.txt, 17379.v4.txt,
17379.v5.txt, 17379.v6.txt, 17379.v8.txt
>
>
> From https://builds.apache.org/job/PreCommit-HBASE-Build/5053/testReport/org.apache.hadoop.hbase.regionserver/TestHRegionWithInMemoryFlush/testWritesWhileGetting/
:
> {code}
> java.io.IOException: java.util.ConcurrentModificationException
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleException(HRegion.java:5886)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5856)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5819)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2786)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2766)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7036)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7015)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6994)
> 	at org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting(TestHRegion.java:4141)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> 	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException: null
> 	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> 	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> 	at org.apache.hadoop.hbase.regionserver.CompactionPipeline.getScanners(CompactionPipeline.java:220)
> 	at org.apache.hadoop.hbase.regionserver.CompactingMemStore.getScanners(CompactingMemStore.java:298)
> 	at org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1154)
> 	at org.apache.hadoop.hbase.regionserver.Store.getScanners(Store.java:97)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.getScannersNoCompaction(StoreScanner.java:353)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:210)
> 	at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:1892)
> 	at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1880)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5842)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5819)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2786)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2766)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7036)
> {code}
> The cause is in CompactionPipeline#getScanners() where there is no synchronization around
iterating pipeline.
> The code causing ConcurrentModificationException:
> {code}
>     for (Segment segment : this.pipeline) {
> {code}
> was introduced by HBASE-17081



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message