Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 689B4200D36 for ; Mon, 6 Nov 2017 12:30:13 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 66F81160BEC; Mon, 6 Nov 2017 11:30:13 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 87BDA1609E0 for ; Mon, 6 Nov 2017 12:30:12 +0100 (CET) Received: (qmail 66824 invoked by uid 500); 6 Nov 2017 11:30:11 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 66813 invoked by uid 99); 6 Nov 2017 11:30:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Nov 2017 11:30:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CFD08180812 for ; Mon, 6 Nov 2017 11:30:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id cLYtcz0Iho_i for ; Mon, 6 Nov 2017 11:30:04 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id D567A5FD7E for ; Mon, 6 Nov 2017 11:30:03 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 80817E0E05 for ; Mon, 6 Nov 2017 11:30:03 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id EC5C6241B6 for ; Mon, 6 Nov 2017 11:30:00 +0000 (UTC) Date: Mon, 6 Nov 2017 11:30:00 +0000 (UTC) From: "Ricardo Bartolome (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-13757) Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 06 Nov 2017 11:30:13 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-13757?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId= =3D16240167#comment-16240167 ]=20 Ricardo Bartolome commented on CASSANDRA-13757: ----------------------------------------------- I have a different problem, but it also is caused by segfaults. I don't thi= nk it's related, but in case you are interested, see CASSANDRA-13663. Are you storing memtables onheap or offheap? See config value *memtable_all= ocation_type*. In our case we are using offheap_buffers > Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running > ---------------------------------------------------------------- > > Key: CASSANDRA-13757 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1375= 7 > Project: Cassandra > Issue Type: Bug > Environment: Operation System: Debian Jessie > Java: Oracle JDK 1.8.0_131 > Cassandra: 3.5.0 > Reporter: Serhat R=C4=B1fat Demircan > > We got following error while repair job running on our cluster. One of th= e nodes stop due to segmantation fault in JVM and repair job fails. > We could not reproduce this problem on our test and staging enviroment (m= ain difference is data size). > {code:java} > # > # SIGSEGV (0xb) at pc=3D0x00007fd80a399e70, pid=3D1305, tid=3D0x00007fd7= ee7c4700 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0= _131-b11) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux= -amd64 compressed oops) > # Problematic frame: > # C [liblz4-java3580121503903465201.so+0x5e70] LZ4_decompress_fast+0xd0 > # > # Failed to write core dump. Core dumps have been disabled. To enable cor= e dumping, try "ulimit -c unlimited" before starting Java again > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > --------------- T H R E A D --------------- > Current thread (0x00007fce32dad1b0): JavaThread "CompactionExecutor:9798= " daemon [_thread_in_native, id=3D16879, stack(0x00007fd7ee784000,0x00007fd= 7ee7c5000)] > siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000= 07fd450c4d000 > Registers: > RAX=3D0x00007fcde6560d32, RBX=3D0x00007fd450c4cff9, RCX=3D0x00007fcde6560= c7a, RDX=3D0x00007fcde6560d3e > RSP=3D0x00007fd7ee7c3160, RBP=3D0x00007fd450c44ae6, RSI=3D0x00007fcde6562= ff8, RDI=3D0x00000000000000c2 > R8 =3D0x00007fcde6562ff4, R9 =3D0x00007fcde6563000, R10=3D0x0000000000000= 000, R11=3D0x0000000000000000 > R12=3D0x000000000000000c, R13=3D0x00007fd4501cd000, R14=3D0x00007fcde6562= ff7, R15=3D0x00007fcde6562ffb > RIP=3D0x00007fd80a399e70, EFLAGS=3D0x0000000000010283, CSGSFS=3D0x0000000= 000000033, ERR=3D0x0000000000000004 > TRAPNO=3D0x000000000000000e > Top of Stack: (sp=3D0x00007fd7ee7c3160) > 0x00007fd7ee7c3160: 0000000000000008 00007fd81e21c3d0 > 0x00007fd7ee7c3170: 0000000000000004 0000000000000001 > 0x00007fd7ee7c3180: 0000000000000002 0000000000000001 > 0x00007fd7ee7c3190: 0000000000000004 0000000000000004 > 0x00007fd7ee7c31a0: 0000000000000004 0000000000000004 > 0x00007fd7ee7c31b0: 0000000000000000 0000000000000000 > 0x00007fd7ee7c31c0: 0000000000000000 ffffffffffffffff > 0x00007fd7ee7c31d0: 0000000000000000 0000000000000001 > 0x00007fd7ee7c31e0: 0000000000000002 0000000000000003 > 0x00007fd7ee7c31f0: 00007fd7ee7c32b8 00007fce32dad3a8 > 0x00007fd7ee7c3200: 0000000000000000 0000000000000000 > 0x00007fd7ee7c3210: 00007fd4501cd000 00007fcde6553000 > 0x00007fd7ee7c3220: 0000000000a77ae6 00007fd80a39659d > 0x00007fd7ee7c3230: 0000000000000000 00000000dcb8fc9b > 0x00007fd7ee7c3240: 00007fd7ee7c32d0 0000000000000000 > 0x00007fd7ee7c3250: 00000006e5c7e4d8 00007fd7ee7c32b8 > 0x00007fd7ee7c3260: 00007fce32dad1b0 00007fd81df2099d > 0x00007fd7ee7c3270: 00007fd7ee7c32a8 0000000000000000 > 0x00007fd7ee7c3280: 0000000000010000 0000000000000000 > 0x00007fd7ee7c3290: 00000006e5c7e528 00007fd81d74df10 > 0x00007fd7ee7c32a0: 0000000000000000 00000006e5c7e4d8 > 0x00007fd7ee7c32b0: 00000006f6c7fbf8 00000006f6e957f0 > 0x00007fd7ee7c32c0: 00000006e5c7e350 00007fd87fffffff > 0x00007fd7ee7c32d0: 00000006e5c7e528 00007fd81fa867e0 > 0x00007fd7ee7c32e0: 00a77ae200010000 0000000000a77ae2 > 0x00007fd7ee7c32f0: 00000006e5c7e488 000000000112d5f1 > 0x00007fd7ee7c3300: dcb8fc9b000099ce 0001000000a77ae6 > 0x00007fd7ee7c3310: 00a814b000a814b4 00000006e5c7e4d8 > 0x00007fd7ee7c3320: 00000006e5c7e4d8 00000006f6a4df38 > 0x00007fd7ee7c3330: 0000000600010000 000000067fffffff > 0x00007fd7ee7c3340: 0089715800002c8a 0006189d87852057 > 0x00007fd7ee7c3350: 0000000000000000 00000000e5244e71 > Instructions: (pc=3D0x00007fd80a399e70) > 0x00007fd80a399e50: e4 0f 49 83 fc 0f 0f 84 94 00 00 00 4a 8d 14 20 > 0x00007fd80a399e60: 48 39 f2 0f 87 c0 00 00 00 0f 1f 80 00 00 00 00 > 0x00007fd80a399e70: 48 8b 0b 48 83 c3 08 48 89 08 48 83 c0 08 48 39 > 0x00007fd80a399e80: c2 77 ed 48 29 d0 48 89 d1 48 29 c3 0f b7 03 48 > Register to memory mapping: > RAX=3D0x00007fcde6560d32 is an unknown value > RBX=3D0x00007fd450c4cff9 is an unknown value > RCX=3D0x00007fcde6560c7a is an unknown value > RDX=3D0x00007fcde6560d3e is an unknown value > RSP=3D0x00007fd7ee7c3160 is pointing into the stack for thread: 0x00007fc= e32dad1b0 > RBP=3D0x00007fd450c44ae6 is an unknown value > RSI=3D0x00007fcde6562ff8 is an unknown value > RDI=3D0x00000000000000c2 is an unknown value > R8 =3D0x00007fcde6562ff4 is an unknown value > R9 =3D0x00007fcde6563000 is an unknown value > R10=3D0x0000000000000000 is an unknown value > R11=3D0x0000000000000000 is an unknown value > R12=3D0x000000000000000c is an unknown value > R13=3D0x00007fd4501cd000 is an unknown value > R14=3D0x00007fcde6562ff7 is an unknown value > R15=3D0x00007fcde6562ffb is an unknown value > Stack: [0x00007fd7ee784000,0x00007fd7ee7c5000], sp=3D0x00007fd7ee7c3160,= free space=3D252k > Native frames: (J=3Dcompiled Java code, j=3Dinterpreted, Vv=3DVM code, C= =3Dnative code) > C [liblz4-java3580121503903465201.so+0x5e70] LZ4_decompress_fast+0xd0 > Java frames: (J=3Dcompiled Java code, j=3Dinterpreted, Vv=3DVM code) > J 5084 net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast([BLjava/nio/ByteBuffer= ;I[BLjava/nio/ByteBuffer;II)I (0 bytes) @ 0x00007fd81df20923 [0x00007fd81df= 20880+0xa3] > J 20311 C2 org.apache.cassandra.io.compress.CompressedRandomAccessReader.= reBufferMmap()V (354 bytes) @ 0x00007fd81fa867e0 [0x00007fd81fa86160+0x680] > J 10706 C2 org.apache.cassandra.io.util.RandomAccessReader.reBuffer()V (7= 8 bytes) @ 0x00007fd81ea59c58 [0x00007fd81ea59b20+0x138] > J 19996 C2 org.apache.cassandra.io.util.SegmentedFile.createReader(J)Lorg= /apache/cassandra/io/util/FileDataInput; (12 bytes) @ 0x00007fd81fa32934 [0= x00007fd81fa32540+0x3f4] > J 20645 C2 org.apache.cassandra.db.columniterator.AbstractSSTableIterator= .(Lorg/apache/cassandra/io/sstable/format/SSTableReader;Lorg/apache/c= assandra/io/util/FileDataInput;Lorg/apache/cassandra/db/DecoratedKey;Lorg/a= pache/cassandra/db/RowIndexEntry;Lorg/apache/cassandra/db/Slices;Lorg/apach= e/cassandra/db/filter/ColumnFilter;Z)V (423 bytes) @ 0x00007fd81fbf8cec [0x= 00007fd81fbf8a20+0x2cc] > J 20644 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBou= nd.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered; (120 bytes) @ 0x0= 0007fd81e61b5d0 [0x00007fd81e61b140+0x490] > J 9597 C2 org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBoun= d.computeNext()Ljava/lang/Object; (5 bytes) @ 0x00007fd81da8dad4 [0x00007fd= 81da8daa0+0x34] > J 3432 C2 org.apache.cassandra.utils.AbstractIterator.hasNext()Z (80 byte= s) @ 0x00007fd81da28f20 [0x00007fd81da28e80+0xa0] > J 19755 C2 org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext= ()Ljava/lang/Object; (9 bytes) @ 0x00007fd81f81d00c [0x00007fd81f81cb60+0x4= ac] > J 21810 C2 org.apache.cassandra.db.rows.UnfilteredRowIterators$Unfiltered= RowMergeIterator.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered; (42= bytes) @ 0x00007fd81ef929fc [0x00007fd81ef927a0+0x25c] > J 15842 C2 org.apache.cassandra.db.SinglePartitionReadCommand.withSSTable= sIterated(Ljava/util/List;Lorg/apache/cassandra/metrics/TableMetrics;)Lorg/= apache/cassandra/db/rows/UnfilteredRowIterator; (71 bytes) @ 0x00007fd81ece= 33b4 [0x00007fd81ece3200+0x1b4] > J 21162 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtab= leAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/c= assandra/db/rows/UnfilteredRowIterator; (635 bytes) @ 0x00007fd81fdb2ff0 [0= x00007fd81fdb2100+0xef0] > J 27194 C2 org.apache.cassandra.db.Keyspace.indexPartition(Lorg/apache/ca= ssandra/db/DecoratedKey;Lorg/apache/cassandra/db/ColumnFamilyStore;Ljava/ut= il/Set;)V (274 bytes) @ 0x00007fd820c8b044 [0x00007fd820c8aae0+0x564] > J 27422 C2 org.apache.cassandra.index.internal.CollatedViewIndexBuilder.b= uild()V (73 bytes) @ 0x00007fd81f0b5a00 [0x00007fd81f0b5940+0xc0] > J 27619 C1 org.apache.cassandra.db.compaction.CompactionManager$12.run()V= (56 bytes) @ 0x00007fd82051313c [0x00007fd820512a00+0x73c] > J 24483 C2 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007fd= 81d50faf4 [0x00007fd81d50fa20+0xd4] > J 23783 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/c= oncurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007fd81fca71a8 [0x= 00007fd81fca70a0+0x108] > J 21390 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007fd81e6d3cb8 [0x000= 07fd81e6d3c60+0x58] > v ~StubRoutines::call_stub > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org For additional commands, e-mail: commits-help@cassandra.apache.org