cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Shuler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13757) Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running
Date Tue, 19 Sep 2017 15:16:01 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171873#comment-16171873
] 

Michael Shuler commented on CASSANDRA-13757:
--------------------------------------------

3.5 has a lot of bugs fixed in subsequent versions up through 3.11.0, so I'd be interested
to know if this reproduces in 3.11.0. I'm also working on 3.11.1 release soon, if you want
to give a try after that has been released.

> Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running
> ----------------------------------------------------------------
>
>                 Key: CASSANDRA-13757
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13757
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Operation System: Debian Jessie
> Java: Oracle JDK 1.8.0_131
> Cassandra: 3.5.0
>            Reporter: Serhat Rıfat Demircan
>
> We got following error while repair job running on our cluster. One of the nodes stop
due to segmantation fault in JVM and repair job fails.
> We could not reproduce this problem on our test and staging enviroment (main difference
is data size).
> {code:java}
> #
> #  SIGSEGV (0xb) at pc=0x00007fd80a399e70, pid=1305, tid=0x00007fd7ee7c4700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed
oops)
> # Problematic frame:
> # C  [liblz4-java3580121503903465201.so+0x5e70]  LZ4_decompress_fast+0xd0
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try
"ulimit -c unlimited" before starting Java again
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> ---------------  T H R E A D  ---------------
> Current thread (0x00007fce32dad1b0):  JavaThread "CompactionExecutor:9798" daemon [_thread_in_native,
id=16879, stack(0x00007fd7ee784000,0x00007fd7ee7c5000)]
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007fd450c4d000
> Registers:
> RAX=0x00007fcde6560d32, RBX=0x00007fd450c4cff9, RCX=0x00007fcde6560c7a, RDX=0x00007fcde6560d3e
> RSP=0x00007fd7ee7c3160, RBP=0x00007fd450c44ae6, RSI=0x00007fcde6562ff8, RDI=0x00000000000000c2
> R8 =0x00007fcde6562ff4, R9 =0x00007fcde6563000, R10=0x0000000000000000, R11=0x0000000000000000
> R12=0x000000000000000c, R13=0x00007fd4501cd000, R14=0x00007fcde6562ff7, R15=0x00007fcde6562ffb
> RIP=0x00007fd80a399e70, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
>   TRAPNO=0x000000000000000e
> Top of Stack: (sp=0x00007fd7ee7c3160)
> 0x00007fd7ee7c3160:   0000000000000008 00007fd81e21c3d0
> 0x00007fd7ee7c3170:   0000000000000004 0000000000000001
> 0x00007fd7ee7c3180:   0000000000000002 0000000000000001
> 0x00007fd7ee7c3190:   0000000000000004 0000000000000004
> 0x00007fd7ee7c31a0:   0000000000000004 0000000000000004
> 0x00007fd7ee7c31b0:   0000000000000000 0000000000000000
> 0x00007fd7ee7c31c0:   0000000000000000 ffffffffffffffff
> 0x00007fd7ee7c31d0:   0000000000000000 0000000000000001
> 0x00007fd7ee7c31e0:   0000000000000002 0000000000000003
> 0x00007fd7ee7c31f0:   00007fd7ee7c32b8 00007fce32dad3a8
> 0x00007fd7ee7c3200:   0000000000000000 0000000000000000
> 0x00007fd7ee7c3210:   00007fd4501cd000 00007fcde6553000
> 0x00007fd7ee7c3220:   0000000000a77ae6 00007fd80a39659d
> 0x00007fd7ee7c3230:   0000000000000000 00000000dcb8fc9b
> 0x00007fd7ee7c3240:   00007fd7ee7c32d0 0000000000000000
> 0x00007fd7ee7c3250:   00000006e5c7e4d8 00007fd7ee7c32b8
> 0x00007fd7ee7c3260:   00007fce32dad1b0 00007fd81df2099d
> 0x00007fd7ee7c3270:   00007fd7ee7c32a8 0000000000000000
> 0x00007fd7ee7c3280:   0000000000010000 0000000000000000
> 0x00007fd7ee7c3290:   00000006e5c7e528 00007fd81d74df10
> 0x00007fd7ee7c32a0:   0000000000000000 00000006e5c7e4d8
> 0x00007fd7ee7c32b0:   00000006f6c7fbf8 00000006f6e957f0
> 0x00007fd7ee7c32c0:   00000006e5c7e350 00007fd87fffffff
> 0x00007fd7ee7c32d0:   00000006e5c7e528 00007fd81fa867e0
> 0x00007fd7ee7c32e0:   00a77ae200010000 0000000000a77ae2
> 0x00007fd7ee7c32f0:   00000006e5c7e488 000000000112d5f1
> 0x00007fd7ee7c3300:   dcb8fc9b000099ce 0001000000a77ae6
> 0x00007fd7ee7c3310:   00a814b000a814b4 00000006e5c7e4d8
> 0x00007fd7ee7c3320:   00000006e5c7e4d8 00000006f6a4df38
> 0x00007fd7ee7c3330:   0000000600010000 000000067fffffff
> 0x00007fd7ee7c3340:   0089715800002c8a 0006189d87852057
> 0x00007fd7ee7c3350:   0000000000000000 00000000e5244e71
> Instructions: (pc=0x00007fd80a399e70)
> 0x00007fd80a399e50:   e4 0f 49 83 fc 0f 0f 84 94 00 00 00 4a 8d 14 20
> 0x00007fd80a399e60:   48 39 f2 0f 87 c0 00 00 00 0f 1f 80 00 00 00 00
> 0x00007fd80a399e70:   48 8b 0b 48 83 c3 08 48 89 08 48 83 c0 08 48 39
> 0x00007fd80a399e80:   c2 77 ed 48 29 d0 48 89 d1 48 29 c3 0f b7 03 48
> Register to memory mapping:
> RAX=0x00007fcde6560d32 is an unknown value
> RBX=0x00007fd450c4cff9 is an unknown value
> RCX=0x00007fcde6560c7a is an unknown value
> RDX=0x00007fcde6560d3e is an unknown value
> RSP=0x00007fd7ee7c3160 is pointing into the stack for thread: 0x00007fce32dad1b0
> RBP=0x00007fd450c44ae6 is an unknown value
> RSI=0x00007fcde6562ff8 is an unknown value
> RDI=0x00000000000000c2 is an unknown value
> R8 =0x00007fcde6562ff4 is an unknown value
> R9 =0x00007fcde6563000 is an unknown value
> R10=0x0000000000000000 is an unknown value
> R11=0x0000000000000000 is an unknown value
> R12=0x000000000000000c is an unknown value
> R13=0x00007fd4501cd000 is an unknown value
> R14=0x00007fcde6562ff7 is an unknown value
> R15=0x00007fcde6562ffb is an unknown value
> Stack: [0x00007fd7ee784000,0x00007fd7ee7c5000],  sp=0x00007fd7ee7c3160,  free space=252k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> C  [liblz4-java3580121503903465201.so+0x5e70]  LZ4_decompress_fast+0xd0
> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> J 5084  net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast([BLjava/nio/ByteBuffer;I[BLjava/nio/ByteBuffer;II)I
(0 bytes) @ 0x00007fd81df20923 [0x00007fd81df20880+0xa3]
> J 20311 C2 org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap()V
(354 bytes) @ 0x00007fd81fa867e0 [0x00007fd81fa86160+0x680]
> J 10706 C2 org.apache.cassandra.io.util.RandomAccessReader.reBuffer()V (78 bytes) @ 0x00007fd81ea59c58
[0x00007fd81ea59b20+0x138]
> J 19996 C2 org.apache.cassandra.io.util.SegmentedFile.createReader(J)Lorg/apache/cassandra/io/util/FileDataInput;
(12 bytes) @ 0x00007fd81fa32934 [0x00007fd81fa32540+0x3f4]
> J 20645 C2 org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(Lorg/apache/cassandra/io/sstable/format/SSTableReader;Lorg/apache/cassandra/io/util/FileDataInput;Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/RowIndexEntry;Lorg/apache/cassandra/db/Slices;Lorg/apache/cassandra/db/filter/ColumnFilter;Z)V
(423 bytes) @ 0x00007fd81fbf8cec [0x00007fd81fbf8a20+0x2cc]
> J 20644 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered;
(120 bytes) @ 0x00007fd81e61b5d0 [0x00007fd81e61b140+0x490]
> J 9597 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Ljava/lang/Object;
(5 bytes) @ 0x00007fd81da8dad4 [0x00007fd81da8daa0+0x34]
> J 3432 C2 org.apache.cassandra.utils.AbstractIterator.hasNext()Z (80 bytes) @ 0x00007fd81da28f20
[0x00007fd81da28e80+0xa0]
> J 19755 C2 org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext()Ljava/lang/Object;
(9 bytes) @ 0x00007fd81f81d00c [0x00007fd81f81cb60+0x4ac]
> J 21810 C2 org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered;
(42 bytes) @ 0x00007fd81ef929fc [0x00007fd81ef927a0+0x25c]
> J 15842 C2 org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(Ljava/util/List;Lorg/apache/cassandra/metrics/TableMetrics;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
(71 bytes) @ 0x00007fd81ece33b4 [0x00007fd81ece3200+0x1b4]
> J 21162 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
(635 bytes) @ 0x00007fd81fdb2ff0 [0x00007fd81fdb2100+0xef0]
> J 27194 C2 org.apache.cassandra.db.Keyspace.indexPartition(Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/ColumnFamilyStore;Ljava/util/Set;)V
(274 bytes) @ 0x00007fd820c8b044 [0x00007fd820c8aae0+0x564]
> J 27422 C2 org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build()V (73
bytes) @ 0x00007fd81f0b5a00 [0x00007fd81f0b5940+0xc0]
> J 27619 C1 org.apache.cassandra.db.compaction.CompactionManager$12.run()V (56 bytes)
@ 0x00007fd82051313c [0x00007fd820512a00+0x73c]
> J 24483 C2 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007fd81d50faf4 [0x00007fd81d50fa20+0xd4]
> J 23783 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(225 bytes) @ 0x00007fd81fca71a8 [0x00007fd81fca70a0+0x108]
> J 21390 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007fd81e6d3cb8 [0x00007fd81e6d3c60+0x58]
> v  ~StubRoutines::call_stub
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message