cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ricardo Bartolome (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13757) Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running
Date Mon, 06 Nov 2017 11:30:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240167#comment-16240167
] 

Ricardo Bartolome commented on CASSANDRA-13757:
-----------------------------------------------

I have a different problem, but it also is caused by segfaults. I don't think it's related,
but in case you are interested, see CASSANDRA-13663.

Are you storing memtables onheap or offheap? See config value *memtable_allocation_type*.
In our case we are using offheap_buffers

> Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running
> ----------------------------------------------------------------
>
>                 Key: CASSANDRA-13757
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13757
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Operation System: Debian Jessie
> Java: Oracle JDK 1.8.0_131
> Cassandra: 3.5.0
>            Reporter: Serhat Rıfat Demircan
>
> We got following error while repair job running on our cluster. One of the nodes stop
due to segmantation fault in JVM and repair job fails.
> We could not reproduce this problem on our test and staging enviroment (main difference
is data size).
> {code:java}
> #
> #  SIGSEGV (0xb) at pc=0x00007fd80a399e70, pid=1305, tid=0x00007fd7ee7c4700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed
oops)
> # Problematic frame:
> # C  [liblz4-java3580121503903465201.so+0x5e70]  LZ4_decompress_fast+0xd0
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try
"ulimit -c unlimited" before starting Java again
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> ---------------  T H R E A D  ---------------
> Current thread (0x00007fce32dad1b0):  JavaThread "CompactionExecutor:9798" daemon [_thread_in_native,
id=16879, stack(0x00007fd7ee784000,0x00007fd7ee7c5000)]
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007fd450c4d000
> Registers:
> RAX=0x00007fcde6560d32, RBX=0x00007fd450c4cff9, RCX=0x00007fcde6560c7a, RDX=0x00007fcde6560d3e
> RSP=0x00007fd7ee7c3160, RBP=0x00007fd450c44ae6, RSI=0x00007fcde6562ff8, RDI=0x00000000000000c2
> R8 =0x00007fcde6562ff4, R9 =0x00007fcde6563000, R10=0x0000000000000000, R11=0x0000000000000000
> R12=0x000000000000000c, R13=0x00007fd4501cd000, R14=0x00007fcde6562ff7, R15=0x00007fcde6562ffb
> RIP=0x00007fd80a399e70, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
>   TRAPNO=0x000000000000000e
> Top of Stack: (sp=0x00007fd7ee7c3160)
> 0x00007fd7ee7c3160:   0000000000000008 00007fd81e21c3d0
> 0x00007fd7ee7c3170:   0000000000000004 0000000000000001
> 0x00007fd7ee7c3180:   0000000000000002 0000000000000001
> 0x00007fd7ee7c3190:   0000000000000004 0000000000000004
> 0x00007fd7ee7c31a0:   0000000000000004 0000000000000004
> 0x00007fd7ee7c31b0:   0000000000000000 0000000000000000
> 0x00007fd7ee7c31c0:   0000000000000000 ffffffffffffffff
> 0x00007fd7ee7c31d0:   0000000000000000 0000000000000001
> 0x00007fd7ee7c31e0:   0000000000000002 0000000000000003
> 0x00007fd7ee7c31f0:   00007fd7ee7c32b8 00007fce32dad3a8
> 0x00007fd7ee7c3200:   0000000000000000 0000000000000000
> 0x00007fd7ee7c3210:   00007fd4501cd000 00007fcde6553000
> 0x00007fd7ee7c3220:   0000000000a77ae6 00007fd80a39659d
> 0x00007fd7ee7c3230:   0000000000000000 00000000dcb8fc9b
> 0x00007fd7ee7c3240:   00007fd7ee7c32d0 0000000000000000
> 0x00007fd7ee7c3250:   00000006e5c7e4d8 00007fd7ee7c32b8
> 0x00007fd7ee7c3260:   00007fce32dad1b0 00007fd81df2099d
> 0x00007fd7ee7c3270:   00007fd7ee7c32a8 0000000000000000
> 0x00007fd7ee7c3280:   0000000000010000 0000000000000000
> 0x00007fd7ee7c3290:   00000006e5c7e528 00007fd81d74df10
> 0x00007fd7ee7c32a0:   0000000000000000 00000006e5c7e4d8
> 0x00007fd7ee7c32b0:   00000006f6c7fbf8 00000006f6e957f0
> 0x00007fd7ee7c32c0:   00000006e5c7e350 00007fd87fffffff
> 0x00007fd7ee7c32d0:   00000006e5c7e528 00007fd81fa867e0
> 0x00007fd7ee7c32e0:   00a77ae200010000 0000000000a77ae2
> 0x00007fd7ee7c32f0:   00000006e5c7e488 000000000112d5f1
> 0x00007fd7ee7c3300:   dcb8fc9b000099ce 0001000000a77ae6
> 0x00007fd7ee7c3310:   00a814b000a814b4 00000006e5c7e4d8
> 0x00007fd7ee7c3320:   00000006e5c7e4d8 00000006f6a4df38
> 0x00007fd7ee7c3330:   0000000600010000 000000067fffffff
> 0x00007fd7ee7c3340:   0089715800002c8a 0006189d87852057
> 0x00007fd7ee7c3350:   0000000000000000 00000000e5244e71
> Instructions: (pc=0x00007fd80a399e70)
> 0x00007fd80a399e50:   e4 0f 49 83 fc 0f 0f 84 94 00 00 00 4a 8d 14 20
> 0x00007fd80a399e60:   48 39 f2 0f 87 c0 00 00 00 0f 1f 80 00 00 00 00
> 0x00007fd80a399e70:   48 8b 0b 48 83 c3 08 48 89 08 48 83 c0 08 48 39
> 0x00007fd80a399e80:   c2 77 ed 48 29 d0 48 89 d1 48 29 c3 0f b7 03 48
> Register to memory mapping:
> RAX=0x00007fcde6560d32 is an unknown value
> RBX=0x00007fd450c4cff9 is an unknown value
> RCX=0x00007fcde6560c7a is an unknown value
> RDX=0x00007fcde6560d3e is an unknown value
> RSP=0x00007fd7ee7c3160 is pointing into the stack for thread: 0x00007fce32dad1b0
> RBP=0x00007fd450c44ae6 is an unknown value
> RSI=0x00007fcde6562ff8 is an unknown value
> RDI=0x00000000000000c2 is an unknown value
> R8 =0x00007fcde6562ff4 is an unknown value
> R9 =0x00007fcde6563000 is an unknown value
> R10=0x0000000000000000 is an unknown value
> R11=0x0000000000000000 is an unknown value
> R12=0x000000000000000c is an unknown value
> R13=0x00007fd4501cd000 is an unknown value
> R14=0x00007fcde6562ff7 is an unknown value
> R15=0x00007fcde6562ffb is an unknown value
> Stack: [0x00007fd7ee784000,0x00007fd7ee7c5000],  sp=0x00007fd7ee7c3160,  free space=252k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> C  [liblz4-java3580121503903465201.so+0x5e70]  LZ4_decompress_fast+0xd0
> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> J 5084  net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast([BLjava/nio/ByteBuffer;I[BLjava/nio/ByteBuffer;II)I
(0 bytes) @ 0x00007fd81df20923 [0x00007fd81df20880+0xa3]
> J 20311 C2 org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap()V
(354 bytes) @ 0x00007fd81fa867e0 [0x00007fd81fa86160+0x680]
> J 10706 C2 org.apache.cassandra.io.util.RandomAccessReader.reBuffer()V (78 bytes) @ 0x00007fd81ea59c58
[0x00007fd81ea59b20+0x138]
> J 19996 C2 org.apache.cassandra.io.util.SegmentedFile.createReader(J)Lorg/apache/cassandra/io/util/FileDataInput;
(12 bytes) @ 0x00007fd81fa32934 [0x00007fd81fa32540+0x3f4]
> J 20645 C2 org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(Lorg/apache/cassandra/io/sstable/format/SSTableReader;Lorg/apache/cassandra/io/util/FileDataInput;Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/RowIndexEntry;Lorg/apache/cassandra/db/Slices;Lorg/apache/cassandra/db/filter/ColumnFilter;Z)V
(423 bytes) @ 0x00007fd81fbf8cec [0x00007fd81fbf8a20+0x2cc]
> J 20644 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered;
(120 bytes) @ 0x00007fd81e61b5d0 [0x00007fd81e61b140+0x490]
> J 9597 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Ljava/lang/Object;
(5 bytes) @ 0x00007fd81da8dad4 [0x00007fd81da8daa0+0x34]
> J 3432 C2 org.apache.cassandra.utils.AbstractIterator.hasNext()Z (80 bytes) @ 0x00007fd81da28f20
[0x00007fd81da28e80+0xa0]
> J 19755 C2 org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext()Ljava/lang/Object;
(9 bytes) @ 0x00007fd81f81d00c [0x00007fd81f81cb60+0x4ac]
> J 21810 C2 org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered;
(42 bytes) @ 0x00007fd81ef929fc [0x00007fd81ef927a0+0x25c]
> J 15842 C2 org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(Ljava/util/List;Lorg/apache/cassandra/metrics/TableMetrics;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
(71 bytes) @ 0x00007fd81ece33b4 [0x00007fd81ece3200+0x1b4]
> J 21162 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
(635 bytes) @ 0x00007fd81fdb2ff0 [0x00007fd81fdb2100+0xef0]
> J 27194 C2 org.apache.cassandra.db.Keyspace.indexPartition(Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/ColumnFamilyStore;Ljava/util/Set;)V
(274 bytes) @ 0x00007fd820c8b044 [0x00007fd820c8aae0+0x564]
> J 27422 C2 org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build()V (73
bytes) @ 0x00007fd81f0b5a00 [0x00007fd81f0b5940+0xc0]
> J 27619 C1 org.apache.cassandra.db.compaction.CompactionManager$12.run()V (56 bytes)
@ 0x00007fd82051313c [0x00007fd820512a00+0x73c]
> J 24483 C2 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007fd81d50faf4 [0x00007fd81d50fa20+0xd4]
> J 23783 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(225 bytes) @ 0x00007fd81fca71a8 [0x00007fd81fca70a0+0x108]
> J 21390 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007fd81e6d3cb8 [0x00007fd81e6d3c60+0x58]
> v  ~StubRoutines::call_stub
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message