Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 60E679862 for ; Wed, 2 Nov 2011 13:04:18 +0000 (UTC) Received: (qmail 17329 invoked by uid 500); 2 Nov 2011 13:04:14 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 17287 invoked by uid 500); 2 Nov 2011 13:04:14 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 17246 invoked by uid 99); 2 Nov 2011 13:04:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Nov 2011 13:04:14 +0000 X-ASF-Spam-Status: No, hits=3.0 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,TRACKER_ID X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of md.jahangir27@gmail.com designates 209.85.161.44 as permitted sender) Received: from [209.85.161.44] (HELO mail-fx0-f44.google.com) (209.85.161.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Nov 2011 13:04:07 +0000 Received: by faas12 with SMTP id s12so486369faa.31 for ; Wed, 02 Nov 2011 06:03:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=vNH3YScIc7jXGkOUfKj4ri7HPHwXUwiiAtnFtIlq5ok=; b=M/bInUVhoqYJMahSW+tbJ+AUdW7q5mlMYzH0WDnST/EHEPtRlbGhzHPny89HgHQg/L tZutRqDCtJ7LhAQSJhgLajqZjVi06WSt12Csq6BaIXDWWg84zdmMcoA3/QBj//IcWnUi pg1ibBIBWel6V0xT52RU3k5wmdPN5tiesSkrU= MIME-Version: 1.0 Received: by 10.223.30.149 with SMTP id u21mr8440937fac.18.1320239026463; Wed, 02 Nov 2011 06:03:46 -0700 (PDT) Received: by 10.152.6.36 with HTTP; Wed, 2 Nov 2011 06:03:46 -0700 (PDT) Date: Wed, 2 Nov 2011 09:03:46 -0400 Message-ID: Subject: 0.8.1: JVM Crash Segmentation Fault From: Jahangir Mohammed To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000e0ce0082830e61704b0c01ac3 --000e0ce0082830e61704b0c01ac3 Content-Type: text/plain; charset=ISO-8859-1 Hello All, JVM is crashing on the cassandra nodes. Re-start doesn't help for long. Ring information: $ bin/nodetool -h A ring; Address DC Rack Status State Load Owns Token 127605887595351923798765477786913079297 A DC1 RAC1 Up Normal 83.65 GB 25.00% 0 B DC2 RAC1 Down Normal 170.09 GB 0.00% 1 C DC1 RAC1 Up Normal 94.6 GB 25.00% 42535295865117307932921825928971026432 D DC2 RAC1 Up Normal 87 GB 0.00% 42535295865117307932921825928971026433 E DC1 RAC1 Up Normal 98.05 GB 25.00% 85070591730234615865843651857942052864 F DC2 RAC1 Up Normal 95.55 GB 0.00% 85070591730234615865843651857942052865 G DC1 RAC1 Up Normal 111.22 GB 25.00% 127605887595351923798765477786913079296 H DC2 RAC1 Up Normal 42.05 GB 0.00% 127605887595351923798765477786913079297 Details: 10GB Heap space. Memory on each node = 98 GB Disk space on each node = 400 GB JVM Crashes with segmentation faults. Have to do frequent re-starts of the nodes. Space on B is 170 GB and is getting CPU bound on re-start. but didn't get added to ring for almost 7 hours now. Java version: java -version java version "1.6.0_24" Java(TM) SE Runtime Environment (build 1.6.0_24-b07) Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) JVM Crash Error log: # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00002abc7ec41fbc, pid=14232, tid=1104185664 # # JRE version: 6.0_24-b07 # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode linux-amd64 compressed oops) # Problematic frame: # V [libjvm.so+0x30ffbc] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x000000004d374000): GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=14243] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000010 Registers: Any ideas/suggestions? Any preferred JVM version? There is nothing in cassandra logs to identify what's going on. Thanks, Jahangir. --000e0ce0082830e61704b0c01ac3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello All,<= div>
JVM is crashing on the cassandra nodes. Re-s= tart doesn't help for long.

Ring inform= ation:
$ bin/nodetool -h A ring;
Address =A0 =A0 =A0 = =A0 DC =A0 =A0 =A0 =A0 =A0Rack =A0 =A0 =A0 =A0Status State =A0 Load =A0 =A0= =A0 =A0 =A0 =A0Owns =A0 =A0Token
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0127605887595351923798765477786913079297
A =A0 DC1 =A0 = =A0 =A0 =A0 RAC1 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A083.65 GB =A0 =A0 =A0 = =A025.00% =A00
B =A0 =A0DC2 =A0 =A0 =A0 =A0 RAC1 =A0 =A0 =A0 =A0D= own =A0 Normal =A0170.09 GB =A0 =A0 =A0 0.00% =A0 1
C =A0 DC1 =A0 =A0 =A0 =A0 RAC1 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A094.= 6 GB =A0 =A0 =A0 =A0 25.00% =A042535295865117307932921825928971026432
=
D =A0 =A0DC2 =A0 =A0 =A0 =A0 RAC1 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A0= 87 GB =A0 =A0 =A0 =A0 =A0 0.00% =A0 42535295865117307932921825928971026433<= /div>
E =A0 DC1 =A0 =A0 =A0 =A0 RAC1 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A098.= 05 GB =A0 =A0 =A0 =A025.00% =A085070591730234615865843651857942052864
=
F =A0 =A0DC2 =A0 =A0 =A0 =A0 RAC1 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A0= 95.55 GB =A0 =A0 =A0 =A00.00% =A0 85070591730234615865843651857942052865
G =A0 DC1 =A0 =A0 =A0 =A0 RAC1 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A0111= .22 GB =A0 =A0 =A0 25.00% =A0127605887595351923798765477786913079296
<= div>H =A0 =A0DC2 =A0 =A0 =A0 =A0 RAC1 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A04= 2.05 GB =A0 =A0 =A0 =A00.00% =A0 127605887595351923798765477786913079297

Details:
10GB Heap space.
Memory on= each node =3D 98 GB
Disk space on each node =3D 400 GB

JVM Crashes with segmentation faults. Have to do frequent r= e-starts of the nodes.
Space on B is 170 GB and is getting CPU bound on re-start. but didn= 9;t get added to ring for almost 7 hours now.

Java= version:
=A0java -version
java version "1.6.0_24&= quot;
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java Ho= tSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

JVM Crash Error log:

#
# A fatal e= rror has been detected by the Java Runtime Environment:
#
# =A0SIGSEGV (0xb) at pc=3D0x00002abc7ec41fbc, pid=3D14232= , tid=3D1104185664
#
# JRE version: 6.0_24-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode linux-a= md64 compressed oops)
# Problematic frame:
# V =A0[libjvm.so+0x30ffbc]
#=
# If you would like to submit a bug report, please visit:
<= div># =A0=A0http://java.sun.com/weba= pps/bugreport/crash.jsp
#

--------------- =A0T H R E A D =A0---------= ------

Current thread (0x000000004d374000): =A0GCT= askThread [stack: 0x0000000000000000,0x0000000000000000] [id=3D14243]
=

siginfo:si_signo=3DSIGSEGV: si_errno=3D0, si_code=3D1 (SEGV_= MAPERR), si_addr=3D0x0000000000000010

Registers:


Any ideas/suggestions? Any preferred= JVM version? There is nothing in cassandra logs to identify what's goi= ng on.

Thanks,
Jahangir.
--000e0ce0082830e61704b0c01ac3--