From user-return-35107-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Jul 8 15:50:48 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AD93F10C0E for ; Mon, 8 Jul 2013 15:50:48 +0000 (UTC) Received: (qmail 4001 invoked by uid 500); 8 Jul 2013 15:50:46 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 3722 invoked by uid 500); 8 Jul 2013 15:50:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 3704 invoked by uid 99); 8 Jul 2013 15:50:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Jul 2013 15:50:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [195.245.230.168] (HELO mail1.bemta3.messagelabs.com) (195.245.230.168) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Jul 2013 15:50:33 +0000 Received: from [195.245.230.131:6862] by server-8.bemta-3.messagelabs.com id 47/F5-18692-4BFDAD15; Mon, 08 Jul 2013 15:50:12 +0000 X-Env-Sender: gdavies@omnifone.com X-Msg-Ref: server-11.tower-78.messagelabs.com!1373298611!30277755!1 X-Originating-IP: [217.156.157.31] X-StarScan-Received: X-StarScan-Version: 6.9.9; banners=omnifone.com,-,- X-VirusChecked: Checked Received: (qmail 6932 invoked from network); 8 Jul 2013 15:50:11 -0000 Received: from unknown (HELO EXFE1IS02.omnifone.com) (217.156.157.31) by server-11.tower-78.messagelabs.com with AES128-SHA encrypted SMTP; 8 Jul 2013 15:50:11 -0000 Received: from EXBE1IS02.omnifone.com ([::1]) by EXFE1IS02.omnifone.com ([::1]) with mapi id 14.01.0438.000; Mon, 8 Jul 2013 16:50:11 +0100 From: Glyn Davies To: "user@cassandra.apache.org" Subject: Re: CassandraDaemon - recent unsafe memory access operation in compiled Java code Thread-Topic: CassandraDaemon - recent unsafe memory access operation in compiled Java code Thread-Index: AQHOeZJ9eQvZybPCfE+Z8K28sg7RBZlZrX2AgAFFAYA= Date: Mon, 8 Jul 2013 15:50:09 +0000 Message-ID: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.1.130117 x-originating-ip: [10.4.1.38] Content-Type: multipart/alternative; boundary="_000_CE009CD71C134gdaviesomnifonecom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CE009CD71C134gdaviesomnifonecom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi, Yes, this continues without the JNA jar. In fact, the only thing which cured it was a reboot (!) Some serious dark magic going on there, as there were no Java processes ru= nning and nothing held the cassandra files open. I found a couple of Java dump texts, and opened an Java bug with one of th= em: http://bugs.sun.com/view_bug.do?bug_id=3D9004953 Though it doesn't seem to show up properly yet Glyn From: sankalp kohli = > Reply-To: "user@cassandra.apache.org" > Date: Sunday, 7 July 2013 22:26 To: "user@cassandra.apache.org" > Subject: Re: CassandraDaemon - recent unsafe memory access operation in co= mpiled Java code have u dropped the JNA jar? Looks like the mmap is failing. On Fri, Jul 5, 2013 at 8:15 AM, Glyn Davies > wrote: Hi, Just starting to experiment with Cassandra, and have hit an early snag. I'm using 1.2.6 on Ubuntu AWS m1.xlarge instances with the Datastax Commun= ity package and have tried using Java versions jdk1.7.0_25 jre1.6.0_45 Also testing with and without libjna-java However, something has triggered a bug in the CassandraDaemon: ERROR [COMMIT-LOG-ALLOCATOR] 2013-07-05 15:00:51,663 CassandraDaemon.java = (line 192) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.lang.InternalError: a fault occurred in a recent unsafe memory access= operation in compiled Java code at org.apache.cassandra.db.commitlog.CommitLogSegment.(Commi= tLogSegment.java:126) at org.apache.cassandra.db.commitlog.CommitLogSegment.freshSegment= (CommitLogSegment.java:81) at org.apache.cassandra.db.commitlog.CommitLogAllocator.createFres= hSegment(CommitLogAllocator.java:250) at org.apache.cassandra.db.commitlog.CommitLogAllocator.access$500= (CommitLogAllocator.java:48) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayTh= row(CommitLogAllocator.java:104) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.= java:28) at java.lang.Thread.run(Unknown Source) This brought two nodes down out of a three node cluster =96 using QUORUM w= rite with 3 replicas. Restarting the node replays this error, so I have the system in a 'stable'= unstable state =96 which is probably a good place for trouble shooting. Presumably something a client wrote triggered this situation, and the othe= r third node was to be the final replication point =96 and is thus still u= p. Any suggestions on next steps? I've had a good google for the error combinations, but didn't find any hit= s. Thanks, Glyn ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ --_000_CE009CD71C134gdaviesomnifonecom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <3BFB00A9F222EE418BCCA7B5252F12FB@omnifone.com> Content-Transfer-Encoding: quoted-printable

Hi,

Yes, this continues without the JNA jar.

In fact, the only thing which cured it was a reboot (!)

Some serious dark magic going on there, as there were no Java process= es running and nothing held the cassandra files open.

I found a couple of Java dump texts, and opened an Java bug with one = of them: http://bugs.sun.com/view_bug.do?bug_id=3D9004953
Though it doesn't seem to show up properly yet

Glyn


From: sankalp kohli <kohlisankalp@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Sunday, 7 July 2013 22:26 To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: CassandraDaemon - rec= ent unsafe memory access operation in compiled Java code

have u dropped the JNA jar? Looks like the mmap is failin= g. 


On Fri, Jul 5, 2013 at 8:15 AM, Glyn Davies <gdavies@omni= fone.com> wrote:


Hi,

Just starting to experiment with Cassandra, and have hit an early sna= g.

I'm using 1.2.6 on Ubuntu AWS m1.xlarge instances with the Datastax C= ommunity package and have tried using Java versions jdk1.7.0_25  jre1= .6.0_45
Also testing with and without libjna-java

However, something has triggered a bug in the CassandraDaemon:

ERROR [COMMIT-LOG-ALLOCATOR] 2013-07-05 15:00:51,663 CassandraDaemon.= java (line 192) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.lang.InternalError: a fault occurred in a recent unsafe memory a= ccess operation in compiled Java code
        at org.apache.cassandra.db.commitlog.Comm= itLogSegment.<init>(CommitLogSegment.java:126)
        at org.apache.cassandra.db.commitlog.Comm= itLogSegment.freshSegment(CommitLogSegment.java:81)
        at org.apache.cassandra.db.commitlog.Comm= itLogAllocator.createFreshSegment(CommitLogAllocator.java:250)
        at org.apache.cassandra.db.commitlog.Comm= itLogAllocator.access$500(CommitLogAllocator.java:48)
        at org.apache.cassandra.db.commitlog.Comm= itLogAllocator$1.runMayThrow(CommitLogAllocator.java:104)
        at org.apache.cassandra.utils.WrappedRunn= able.run(WrappedRunnable.java:28)
        at java.lang.Thread.run(Unknown Source)

This brought two nodes down out of a three node cluster =96 using QUO= RUM write with 3 replicas.
Restarting the node replays this error, so I have the system in a 'st= able' unstable state =96 which is probably a good place for trouble shooti= ng.

Presumably something a client wrote triggered this situation, and the= other third node was to be the final replication point =96 and is thus st= ill up.

Any suggestions on next steps?
I've had a good google for the error combinations, but didn't find an= y hits.

Thanks,

Glyn

______________________________________________________________________
= This email has been scanned by the Symantec Email Security.cloud service.<= br> For more information please visit http://www.symanteccloud.com
______________________________________________________________________
=


______________________________________________________________________
= This email has been scanned by the Symantec Email Security.cloud service.<= br> For more information please visit http://www.symanteccloud.com
______________________________________________________________________
=

______________________________________________________________________
= This email has been scanned by the Symantec Email Security.cloud service.<= BR> For more information please visit http://www.symanteccloud.com
______________________________________________________________________
= --_000_CE009CD71C134gdaviesomnifonecom_--