cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kurt Greaves (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11200) CompactionExecutor thread error brings down JVM in 3.0.3
Date Wed, 08 Feb 2017 11:03:42 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15857837#comment-15857837
] 

Kurt Greaves commented on CASSANDRA-11200:
------------------------------------------

Saw this on a 3.7 node. The IAE exception occured several times prior to the SIGBUS. On restart
there were a couple corrupt SSTables in sstable_activity, not sure if it was related. I don't
have the core dumps but might try see if I can reproduce with some user defined compactions.
{code}
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: ERROR o.a.c.service.CassandraDaemon
Exception in thread Thread[CompactionExecutor:86114,1,main]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]: java.lang.IllegalArgumentException:
null
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at java.nio.Buffer.position(Buffer.java:244)
~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:163)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.util.CompressedSegmentedFile$Mmap.readChunk(CompressedSegmentedFile.java:323)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:137)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
~[caffeine-2.2.6.jar:na]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:220)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:186)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$200(BigTableScanner.java:51)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:285)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:265)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:245)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
~[apache-cassandra-3.7.2.jar:3.7.2]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
Feb 08 08:46:28 ip-10-224-42-94.ec2.internal cassandra[31038]:         at java.lang.Thread.run(Thread.java:745)
[na:1.8.0_111]
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: #
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # A fatal error has been detected
by the Java Runtime Environment:
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: #
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: #  SIGBUS (0x7) at pc=0x00007ff4150520d0,
pid=1, tid=0x00007ff175b28700
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: #
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # JRE version: Java(TM) SE
Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # Java VM: Java HotSpot(TM)
64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # Problematic frame:
Feb 08 08:46:43 ip-10-224-42-94.ec2.internal cassandra[31038]: # v  ~StubRoutines::jbyte_disjoint_arraycopy
{code}

> CompactionExecutor thread error brings down JVM in 3.0.3
> --------------------------------------------------------
>
>                 Key: CASSANDRA-11200
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11200
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: debian jesse latest release, updated Feb. 20th
>            Reporter: Jason Kania
>            Priority: Critical
>
> When launching Cassandra 3.0.3, with java version "1.8.0_74", Cassandra writes the following
to the debug file before a segmentation fault occurs bringing down the JVM - the problem is
repeatable.
> DEBUG [CompactionExecutor:1] 2016-02-20 18:26:16,892 CompactionTask.java:146 - Compacting
(56f677c0-d829-11e5-b23a-25dbd4d727f6) [/var/lib/cassandra/data/sensordb/periodicReading/ma-367-big-Data.db:level=0,
/var/lib/cassandra/data/sensordb/periodicReading/ma-368-big-Data.db:level=0, /var/lib/cassandra/data/sensordb/periodicReading/ma-371-big-Data.db:level=0,
/var/lib/cassandra/data/sensordb/periodicReading/ma-370-big-Data.db:level=0, /var/lib/cassandra/data/sensordb/periodicReading/ma-369-big-Data.db:level=0,
]
> The JVM error that occurs is the following:
> \#
> \# A fatal error has been detected by the Java Runtime Environment:
> \#
> \#  SIGBUS (0x7) at pc=0x00007fa8a1052150, pid=12179, tid=140361951868672
> \#
> \# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
> \# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64 compressed
oops)
> \# Problematic frame:
> \# v  ~StubRoutines::jbyte_disjoint_arraycopy
> \#
> \# Core dump written. Default location: /tmp/core or core.12179
> \#
> \# If you would like to submit a bug report, please visit:
> \#   http://bugreport.java.com/bugreport/crash.jsp
> \#
> ---------------  T H R E A D  ---------------
> Current thread (0x00007fa89c56ac20):  JavaThread "CompactionExecutor:1" daemon [_thread_in_Java,
id=12323, stack(0x00007fa89043f000,0x00007fa890480000)]
> siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007fa838988002
> Even if all of the files associated with "ma-[NNN]*" are removed, the JVM dies with the
same error after the next group of "ma-[NNN]*" are eventually written out and compacted.
> Though this may be strictly a JVM problem, I have seen the issue in Oracle JVM 8.0_65
and 8.0_74 and I raise it in case this problem is due to JNI usage of an external compression
library or some direct memory usage.
> I have a core dump if that is helpful to anyone.
> Bug CASSANDRA-11201 may also be related although when the exception referenced in the
bug occurs, the JVM remains alive.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message