cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tobias Schlottke (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8851) Uncaught exception on thread Thread[SharedPool-Worker-16,5,main] after upgrade to 2.1.3
Date Mon, 16 Mar 2015 07:38:43 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362889#comment-14362889
] 

Tobias Schlottke commented on CASSANDRA-8851:
---------------------------------------------

Hey [~benedict]!
I recently noticed that even with your patch, there are still some exceptions of that kind
happening. Example:

{code}
ERROR [CompactionExecutor:3] 2015-03-16 08:34:24,798 CassandraDaemon.java:167 - Exception
in thread Thread[CompactionExecutor:3,1,main]
java.lang.NullPointerException: null
	at org.apache.cassandra.io.sstable.SSTableReader.cloneWithNewStart(SSTableReader.java:892)
~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:328) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:178)
~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:117) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:192)
~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:72)
~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:240)
~[apache-cassandra-2.1.3.jar:2.1.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_45]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_45]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
	at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
ERROR [Reference-Reaper:1] 2015-03-16 08:34:27,210 Ref.java:181 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@24b4e762) to class org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1623385537:/data/cassandra/data/metrigo_prod/new_user_data/metrigo_prod-new_user_data-jb-164099
was not released before the reference was garbage collected
{code}

Any ideas?

Best,

Tobias

> Uncaught exception on thread Thread[SharedPool-Worker-16,5,main] after upgrade to 2.1.3
> ---------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-8851
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8851
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: ubuntu 
>            Reporter: Tobias Schlottke
>            Assignee: Benedict
>            Priority: Critical
>             Fix For: 2.1.4
>
>
> Hi there,
> after upgrading to 2.1.3 we've got the following error every few seconds:
> {code}
> WARN  [SharedPool-Worker-16] 2015-02-23 10:20:36,392 AbstractTracingAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-16,5,main]: {}
> java.lang.AssertionError: null
> 	at org.apache.cassandra.io.util.Memory.size(Memory.java:307) ~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.utils.obs.OffHeapBitSet.capacity(OffHeapBitSet.java:61) ~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.utils.BloomFilter.indexes(BloomFilter.java:74) ~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.utils.BloomFilter.isPresent(BloomFilter.java:98) ~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:1366)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:1350)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:41)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:185)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:273)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1915)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1748)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:342) ~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:57)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47) ~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_45]
> 	at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.3.jar:2.1.3]
> 	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.3.jar:2.1.3]
> 	at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> {code}
> This seems to crash the compactions and pushes up server load and piles up compactions.
> Any idea / possible workaround?
> Best,
> Tobias



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message