cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Lohfink <clohfin...@gmail.com>
Subject Re: Infinite loop in SliceQueryFilter
Date Fri, 04 Dec 2015 20:33:36 GMT
May just be going over a lot of data. Does output of 'nodetool cfstats'
show large partitions? (partition maximum bytes). "collecting 1 of 2147483647"
is suspicious. Are your queries using ALLOW FILTERING or have very high
limits? If trying to read 2 billion entries in 1 query you will have memory
issues. May want to check with jvmtop/htop to make sure its not GCs using
CPU as well. Is there a sane amount of sstables? Providing some more
details can help (cfstats, cfhistograms, queries your making, schema)

Chris

On Fri, Dec 4, 2015 at 10:43 AM, Xihui He <xihuihe@gmail.com> wrote:

> Dear All,
>
> Recently one of node in our cluster has high cpu load ~100%. It seems to
> me there is a infinite loop in SliceQueryFilter.
>
> The below log is repeated in 5000ms (range_request_timeout_in_ms).
> TRACE [SharedPool-Worker-11] 2015-12-04 19:25:33,418
> SliceQueryFilter.java:269 - collecting 1 of 2147483647:
> images:b608719e728d11e5812b57f4c5416142:false:62@1444838867382000
>
> Our version is 2.19. Here is the bt:
>
> org.apache.cassandra.db.composites.AbstractComposite.isEmpty(AbstractComposite.java:30)
>
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:76)
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
>
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37)
>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
> org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173)
> org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156)
>
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
>
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125)
>
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:264)
>
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:108)
>
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82)
>
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
>
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314)
>
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:2033)
>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1876)
> org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:357)
>
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47)
>
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>
> java.lang.Thread.run(Thread.java:745)
>
> Appreciate if anyone could help.
>
> Thanks in advance,
> Xihui
>

Mime
View raw message