jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Francesco Mari (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (OAK-7755) deadlock TarMK flush, lucene
Date Tue, 18 Sep 2018 09:40:00 GMT

     [ https://issues.apache.org/jira/browse/OAK-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Francesco Mari reassigned OAK-7755:
-----------------------------------

    Assignee: Francesco Mari

> deadlock TarMK flush, lucene
> ----------------------------
>
>                 Key: OAK-7755
>                 URL: https://issues.apache.org/jira/browse/OAK-7755
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segment-tar
>    Affects Versions: 1.6.1
>         Environment: apache sling9, oak 1.6.1
>            Reporter: Guido Dunker
>            Assignee: Francesco Mari
>            Priority: Blocker
>              Labels: deadlock
>
> We are getting the following deadlock. Please help! (Production environment)
> I have already annotated possible locks and synchronized blocks in between:
> {noformat}
> "TarMK flush [/opt/condat/epet9/sling/repository/segmentstore]":
>   waiting to lock Monitor@0x00007fedfc00cc28 (Object@0x00000004795519a8, a org/apache/jackrabbit/oak/segment/SegmentId),
>   which is held by "oak-lucene-14"
> "oak-lucene-14":
>  waiting for ownable synchronizer 0x00000003c13818c0, (a java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync),
>  which is held by "TarMK flush [/opt/condat/epet9/sling/repository/segmentstore]"
> Thread 28883: (state = BLOCKED)
>  - org.apache.jackrabbit.oak.segment.SegmentId.getSegment() @bci=12, line=121 (Compiled
frame)
>         synchronized (this) 
>  - org.apache.jackrabbit.oak.segment.Record.getSegment() @bci=4, line=70 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.BlockRecord.read(int, byte[], int, int) @bci=49,
line=57 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentStream.read(byte[], int, int) @bci=314,
line=189 (Compiled frame)
>  - com.google.common.io.ByteStreams.read(java.io.InputStream, byte[], int, int) @bci=43,
line=828 (Compiled frame)
>  - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[], int, int)
@bci=4, line=695 (Compiled frame)
>  - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[]) @bci=5, line=676
(Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentStream.getString() @bci=93, line=103 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.Segment.readString(int) @bci=189, line=524 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(org.apache.jackrabbit.oak.segment.Segment,
int) @bci=15, line=212 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBlob.readBlobId(org.apache.jackrabbit.oak.segment.Segment,
int) @bci=37, line=167 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.AbstractFileStore$4.consume(int, org.apache.jackrabbit.oak.segment.RecordType,
int) @bci=24, line=354 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.Segment.forEachRecord(org.apache.jackrabbit.oak.segment.Segment$RecordConsumer)
@bci=48, line=716 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.AbstractFileStore.populateTarBinaryReferences(org.apache.jackrabbit.oak.segment.Segment,
org.apache.jackrabbit.oak.segment.file.TarWriter) @bci=25, line=349 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore.writeSegment(org.apache.jackrabbit.oak.segment.SegmentId,
byte[], int, int) @bci=136, line=657 (Compiled frame)
>         fileStoreLock.writeLock().lock(); Zeile 639
>         bis: populateTarBinaryReferences
>  - org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush() @bci=383, line=383
(Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.flush() @bci=165, line=148
(Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentWriter.flush() @bci=4, line=143 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore$7.call() @bci=7, line=373 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore$7.call() @bci=1, line=370 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.file.TarRevisions.doFlush(java.util.concurrent.Callable)
@bci=25, line=224 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.TarRevisions.flush(java.util.concurrent.Callable)
@bci=42, line=212 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore.flush() @bci=20, line=370 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore$2.run() @bci=15, line=233 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.file.SafeRunnable.run() @bci=21, line=67 (Compiled
frame)
>  - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled
frame)
>  - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
@bci=1, line=180 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37,
line=294 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted
frame)
>  - java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
> Locked ownable synchronizers:
>     - <0x00000003c1361228>, (a java/util/concurrent/locks/ReentrantLock$NonfairSync)
>     - <0x00000003c13818c0>, (a java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync)
>     - <0x00000003c1c3a0d8>, (a java/util/concurrent/ThreadPoolExecutor$Worker)
> Thread 31035: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1,
line=836 (Compiled frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int) @bci=83,
line=967 (Interpreted frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int) @bci=10,
line=1283 (Compiled frame)
>  - java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5, line=727
(Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore$8.call() @bci=158, line=567 (Compiled
frame)
>     fileStoreLock.readLock().lock();
>  - org.apache.jackrabbit.oak.segment.file.FileStore$8.call() @bci=1, line=542 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(org.apache.jackrabbit.oak.segment.SegmentId,
java.util.concurrent.Callable) @bci=1, line=95 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(org.apache.jackrabbit.oak.segment.SegmentId)
@bci=14, line=542 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentId.getSegment() @bci=38, line=125 (Compiled
frame)
>     synchronized (this) 
>  - org.apache.jackrabbit.oak.segment.Record.getSegment() @bci=4, line=70 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.BlockRecord.read(int, byte[], int, int) @bci=49,
line=57 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentStream.read(byte[], int, int) @bci=314,
line=189 (Compiled frame)
>  - com.google.common.io.ByteStreams.read(java.io.InputStream, byte[], int, int) @bci=64,
line=833 (Compiled frame)
>  - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[], int, int)
@bci=4, line=695 (Compiled frame)
>  - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[]) @bci=5, line=676
(Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentStream.getString() @bci=93, line=103 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.Segment.readString(int) @bci=189, line=524 (Compiled
frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(org.apache.jackrabbit.oak.segment.Segment,
int) @bci=15, line=212 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBlob.length() @bci=124, line=115 (Compiled
frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.<init>(java.lang.String,
org.apache.jackrabbit.oak.spi.state.NodeBuilder, java.lang.String, org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$BlobFactory)
@bci=204, line=409 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.<init>(java.lang.String,
org.apache.jackrabbit.oak.spi.state.NodeBuilder, java.lang.String, org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$BlobFactory)
@bci=25, line=589 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory.fileLength(java.lang.String)
@bci=64, line=176 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.copyFilesToLocal(org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory$CORFileReference,
boolean, boolean) @bci=195, line=214 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.prefetchIndexFiles()
@bci=96, line=170 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.<init>(org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier,
org.apache.lucene.store.Directory, org.apache.lucene.store.Directory, boolean, java.lang.String,
java.util.concurrent.Executor) @bci=85, line=81 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.wrapForRead(java.lang.String,
org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition, org.apache.lucene.store.Directory,
java.lang.String) @bci=35, line=122 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReader(org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition,
org.apache.jackrabbit.oak.spi.state.NodeState, java.lang.String, java.lang.String, java.lang.String)
@bci=61, line=102 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReaders(org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition,
org.apache.jackrabbit.oak.spi.state.NodeState, java.lang.String) @bci=20, line=61 (Compiled
frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.plugins.index.lucene.reader.LuceneIndexReaderFactory,
org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndexFactory) @bci=17, line=68 (Compiled
frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker$1.leave(org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeState) @bci=30, line=132 (Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(java.lang.String,
org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState)
@bci=66, line=153 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.MapRecord.compare(org.apache.jackrabbit.oak.segment.MapRecord,
org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=197, line=415 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=909, line=608 (Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(java.lang.String,
org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState)
@bci=43, line=148 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.MapRecord.compare(org.apache.jackrabbit.oak.segment.MapRecord,
org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=400, line=457 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=909, line=608 (Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(org.apache.jackrabbit.oak.spi.commit.Editor,
org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState)
@bci=34, line=52 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.diffAndUpdate(org.apache.jackrabbit.oak.spi.state.NodeState)
@bci=140, line=142 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(org.apache.jackrabbit.oak.spi.state.NodeState)
@bci=36, line=113 (Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call() @bci=79, line=135
(Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call() @bci=1, line=128
(Compiled frame)
>  - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
> Locked ownable synchronizers:
>     - <0x0000000476194f30>, (a java/util/concurrent/ThreadPoolExecutor$Worker)   
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message