cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nate McCall <n...@thelastpickle.com>
Subject Re: hanging validation compaction
Date Thu, 13 Apr 2017 09:35:34 GMT
Not sure what is going on there either. Roland - can you open an issue with
the information above:
https://issues.apache.org/jira/browse/CASSANDRA

On Thu, Apr 13, 2017 at 7:49 PM, benjamin roth <brstgt@gmail.com> wrote:

> What I can tell you from that trace - given that this is the correct
> thread and it really hangs there:
>
> The validation is stuck when reading from an SSTable.
> Unfortunately I am no caffeine expert. It looks like the read is cached
> and after the read caffeine tries to drain the cache and this is stuck. I
> don't see the reason from that stack trace.
> Someone had to dig deeper into caffeine to find the root cause.
>
> 2017-04-13 9:27 GMT+02:00 Roland Otta <Roland.Otta@willhaben.at>:
>
>> i had a closer look at the validation executor thread (i hope thats what
>> you meant)
>>
>> it seems the thread is always repeating stuff in
>> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebu
>> ffer(ChunkCache.java:235)
>>
>> here is the full stack trace ...
>>
>> i am sorry .. but i have no clue whats happening there ..
>>
>> com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$64/
>> 2098345091 <(209)%20834-5091>.accept(Unknown Source)
>> com.github.benmanes.caffeine.cache.BoundedBuffer$RingBuffer.
>> drainTo(BoundedBuffer.java:104)
>> com.github.benmanes.caffeine.cache.StripedBuffer.drainTo(Str
>> ipedBuffer.java:160)
>> com.github.benmanes.caffeine.cache.BoundedLocalCache.drainRe
>> adBuffer(BoundedLocalCache.java:964)
>> com.github.benmanes.caffeine.cache.BoundedLocalCache.mainten
>> ance(BoundedLocalCache.java:918)
>> com.github.benmanes.caffeine.cache.BoundedLocalCache.perform
>> CleanUp(BoundedLocalCache.java:903)
>> com.github.benmanes.caffeine.cache.BoundedLocalCache$Perform
>> CleanupTask.run(BoundedLocalCache.java:2680)
>> com.google.common.util.concurrent.MoreExecutors$DirectExecut
>> or.execute(MoreExecutors.java:457)
>> com.github.benmanes.caffeine.cache.BoundedLocalCache.schedul
>> eDrainBuffers(BoundedLocalCache.java:875)
>> com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRe
>> ad(BoundedLocalCache.java:748)
>> com.github.benmanes.caffeine.cache.BoundedLocalCache.compute
>> IfAbsent(BoundedLocalCache.java:1783)
>> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsen
>> t(LocalCache.java:97)
>> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(Loc
>> alLoadingCache.java:66)
>> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebu
>> ffer(ChunkCache.java:235)
>> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebu
>> ffer(ChunkCache.java:213)
>> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(R
>> andomAccessReader.java:65)
>> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(Ran
>> domAccessReader.java:59)
>> org.apache.cassandra.io.util.RebufferingInputStream.read(Reb
>> ufferingInputStream.java:88)
>> org.apache.cassandra.io.util.RebufferingInputStream.readFull
>> y(RebufferingInputStream.java:66)
>> org.apache.cassandra.io.util.RebufferingInputStream.readFull
>> y(RebufferingInputStream.java:60)
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
>> org.apache.cassandra.db.marshal.AbstractType.readValue(
>> AbstractType.java:420)
>> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245)
>> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimple
>> Column(UnfilteredSerializer.java:610)
>> org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$des
>> erializeRowBody$1(UnfilteredSerializer.java:575)
>> org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$84/898489541.accept(Unknown
>> Source)
>> org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1222)
>> org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1177)
>> org.apache.cassandra.db.Columns.apply(Columns.java:377)
>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializ
>> eRowBody(UnfilteredSerializer.java:571)
>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializ
>> e(UnfilteredSerializer.java:440)
>> org.apache.cassandra.io.sstable.SSTableSimpleIterator$Curren
>> tFormatIterator.computeNext(SSTableSimpleIterator.java:95)
>> org.apache.cassandra.io.sstable.SSTableSimpleIterator$Curren
>> tFormatIterator.computeNext(SSTableSimpleIterator.java:73)
>> org.apache.cassandra.utils.AbstractIterator.hasNext(Abstract
>> Iterator.java:47)
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasN
>> ext(SSTableIdentityIterator.java:122)
>> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowI
>> terator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowI
>> terator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>> org.apache.cassandra.utils.AbstractIterator.hasNext(Abstract
>> Iterator.java:47)
>> org.apache.cassandra.utils.MergeIterator$Candidate.advance(
>> MergeIterator.java:374)
>> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(
>> MergeIterator.java:186)
>> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNe
>> xt(MergeIterator.java:155)
>> org.apache.cassandra.utils.AbstractIterator.hasNext(Abstract
>> Iterator.java:47)
>> org.apache.cassandra.db.rows.UnfilteredRowIterators$Unfilter
>> edRowMergeIterator.computeNext(UnfilteredRowIterators.java:500)
>> org.apache.cassandra.db.rows.UnfilteredRowIterators$Unfilter
>> edRowMergeIterator.computeNext(UnfilteredRowIterators.java:360)
>> org.apache.cassandra.utils.AbstractIterator.hasNext(Abstract
>> Iterator.java:47)
>> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
>> org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(U
>> nfilteredRowIterators.java:178)
>> org.apache.cassandra.repair.Validator.rowHash(Validator.java:221)
>> org.apache.cassandra.repair.Validator.add(Validator.java:160)
>> org.apache.cassandra.db.compaction.CompactionManager.doValid
>> ationCompaction(CompactionManager.java:1364)
>> org.apache.cassandra.db.compaction.CompactionManager.access$
>> 700(CompactionManager.java:85)
>> org.apache.cassandra.db.compaction.CompactionManager$13.
>> call(CompactionManager.java:933)
>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$
>> threadLocalDeallocator$0(NamedThreadFactory.java:79)
>> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1371495133.run(Unknown
>> Source)
>> java.lang.Thread.run(Thread.java:745)
>>
>> On Thu, 2017-04-13 at 08:47 +0200, benjamin roth wrote:
>>
>> You should connect to the node with JConsole and see where the compaction
>> thread is stuck
>>
>> 2017-04-13 8:34 GMT+02:00 Roland Otta <Roland.Otta@willhaben.at>:
>>
>> hi,
>>
>> we have the following issue on our 3.10 development cluster.
>>
>> we are doing regular repairs with thelastpickle's fork of creaper.
>> sometimes the repair (it is a full repair in that case) hangs because
>> of a stuck validation compaction
>>
>> nodetool compactionstats gives me
>> a1bb45c0-1fc6-11e7-81de-0fb0b3f5a345 Validation      bds      ad_event
>> 805955242 841258085 bytes 95.80%
>> we have here no more progress for hours
>>
>> nodetool tpstats shows
>> alidationExecutor                1         1          16186         0
>>                0
>>
>> i checked the logs on the affected node and could not find any
>> suspicious errors.
>>
>> anyone that already had this issue and knows how to cope with that?
>>
>> a restart of the node helps to finish the repair ... but i am not sure
>> whether that somehow breaks the full repair
>>
>> bg,
>> roland
>>
>>
>>
>


-- 
-----------------
Nate McCall
Wellington, NZ
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com

Mime
View raw message