Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3B94098FC for ; Thu, 5 Apr 2012 09:55:00 +0000 (UTC) Received: (qmail 22681 invoked by uid 500); 5 Apr 2012 09:54:57 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 22657 invoked by uid 500); 5 Apr 2012 09:54:57 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 22648 invoked by uid 99); 5 Apr 2012 09:54:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Apr 2012 09:54:57 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a47.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Apr 2012 09:54:51 +0000 Received: from homiemail-a47.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTP id DE705284058 for ; Thu, 5 Apr 2012 02:54:29 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=xDlu5u6zC3 BgAtj0vTQhEPdZ80hX8usV9pDrGe2bpb3g9WpeqWBcxW0p5KRU8lyDxJqJgd5Buz vQ9Pz0oQKAh+J+zhNKi6je70jRqbvJQzuQeWrheTXM3tjoH8nU/w3+7aMrzd51Os cW+mZ3lFdGQMiLINol7vl8sGfjfh8mswI= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=xn4YCqyyLenq7rnv 3kdkoIQ0pwM=; b=S7MVhkyCEpljfdFYCA44pyZFN6Lg5OIiWfPqf/Y1XSj+v+3K V5bZxBrlor6EZf3q0TdulgpTafM3m70X0fQ4ov/rKTLbJw3tkfZqPZwdoZ0rR1Ic N1L31HXpOeKINwrm7V9D0FbRYVd//G3YLFoUJFVc84OX3LZVi2NXjpCWw7Y= Received: from [172.16.1.3] (125-236-193-159.adsl.xtra.co.nz [125.236.193.159]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTPSA id 45216284057 for ; Thu, 5 Apr 2012 02:54:27 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: multipart/alternative; boundary="Apple-Mail=_472F8E16-3115-46FB-8CFF-0649952E1BF4" Subject: Re: server down Date: Thu, 5 Apr 2012 21:54:25 +1200 In-Reply-To: To: user@cassandra.apache.org References: <003001cd1256$c9423820$5bc6a860$@com> <49AA9395-02A8-4EE5-93D3-549196C2FF16@thelastpickle.com> Message-Id: X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_472F8E16-3115-46FB-8CFF-0649952E1BF4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Sun or Open JDK ?=20 Either way I would suggest upgrading to the latest JDK, upgrading = cassandra to 1.0.8 and running nodetool upgradetables.=20 If the fault persists after that I would look at IO or memory issues.=20 Hope that helps.=20 ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 5/04/2012, at 4:31 PM, Michael Vaknin wrote: > i am usuinc cassandra 1.0.3 java 6.0.17 on ubuntu 10.04 > this is a stable version for 6 months now. >=20 > On Wed, Apr 4, 2012 at 9:51 PM, aaron morton = wrote: > What version of cassandra are you using ?=20 >=20 > What java vendor / version ?=20 >=20 > What OS vendor / version ?=20 >=20 > Cheers >=20 > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com >=20 > On 4/04/2012, at 11:33 PM, Michael Vaknine wrote: >=20 >> I am recently having problems with 1 of my 4ty cluster servers. >> I would appreciate any help >> =20 >> ERROR 11:23:14,331 Fatal exception in thread = Thread[MutationStage:19,5,main] >> 225 java.lang.InternalError: a fault occurred in a recent unsafe = memory access operation in compiled Java code >> 226 at = org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.j= ava:196) >> 227 at = org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.ja= va:348) >> 228 at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java= :120) >> 229 at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java= :83) >> 230 at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java= :73) >> 231 at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java= :37) >> 232 at = org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(Simpl= eSliceReader.java:79) >> 233 at = org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(Simpl= eSliceReader.java:40) >> 234 at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterat= or.java:140) >> 235 at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:1= 35) >> 236 at = org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTabl= eSliceIterator.java:107) >> 237 at = org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.j= ava:145) >> 238 at = org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.j= ava:124) >> 239 at = org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterat= or.java:98) >> 240 at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterat= or.java:140) >> 241 at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:1= 35) >> 242 at = org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(Slic= eQueryFilter.java:116) >> 243 at = org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java= :144) >> 244 at = org.apache.cassandra.db.CollationController.collectAllData(CollationContro= ller.java:227) >> 245 at = org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCo= ntroller.java:62) >> 246 at = org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyS= tore.java:1274) >> 247 at = org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:= 1141) >> 248 at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStor= e.java:1170) >> 249 at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStor= e.java:1122) >> 250 at = org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:504) >> 251 at org.apache.cassandra.db.Table.apply(Table.java:441) >> 252 at = org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:3= 38) >> 253 at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> 254 at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> 255 at = java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> 256 at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> 257 at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886) >> 258 at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908) >> 259 at java.lang.Thread.run(Thread.java:619) >> 260 INFO 11:23:17,640 Finished reading = /var/lib/cassandra/commitlog/CommitLog-1333465745624.log >> 261 ERROR 11:23:17,641 Exception encountered during startup >> 262 java.lang.RuntimeException: = java.util.concurrent.ExecutionException: java.lang.InternalError: a = fault occurred in a recent unsafe memory access operation in compiled = Java code >> 263 at = org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:508)= >> =20 >> Thanks >> Michael >=20 >=20 --Apple-Mail=_472F8E16-3115-46FB-8CFF-0649952E1BF4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 Sun = or Open JDK ? 

Either way I would suggest = upgrading to the latest JDK, upgrading cassandra to 1.0.8 and running = nodetool upgradetables. 

If the fault = persists after that I would look at IO or memory = issues. 

Hope that = helps. 

http://www.thelastpickle.com

On 5/04/2012, at 4:31 PM, Michael Vaknin wrote:

i am usuinc cassandra 1.0.3 java 6.0.17 on ubuntu = 10.04
this is a stable version for 6 months now.

On Wed, Apr 4, 2012 at 9:51 PM, aaron morton <aaron@thelastpickle.com> wrote:
What version of cassandra are you using = ? 

What java vendor / version ? 

What OS vendor / version = ? 

Cheers

-----------------
Aaron Morton
Freelance = Developer
@aaronmorton

On 4/04/2012, at 11:33 PM, Michael Vaknine = wrote:

I am recently = having problems with 1 of my 4ty cluster = servers.
I would appreciate any help
 <= /div>
ERROR 11:23:14,331 Fatal exception in thread = Thread[MutationStage:19,5,main]
225 java.lang.InternalError: a fault occurred in a recent unsafe memory = access operation in compiled Java code
226     at = org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.j= ava:196)
227     at = org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.ja= va:348)
228     at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java= :120)
229     at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java= :83)
230     at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java= :73)
231     at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java= :37)
232     at = org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(Simpl= eSliceReader.java:79)
233     at = org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(Simpl= eSliceReader.java:40)
234     at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterat= or.java:140)
235     at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:1= 35)
236     at = org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTabl= eSliceIterator.java:107)
237     at = org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.j= ava:145)
238     at = org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.j= ava:124)
239     at = org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterat= or.java:98)
240     at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterat= or.java:140)
241     at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:1= 35)
242     at = org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(Slic= eQueryFilter.java:116)
243     at = org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java= :144)
244     at = org.apache.cassandra.db.CollationController.collectAllData(CollationContro= ller.java:227)
245     at = org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCo= ntroller.java:62)
246     at = org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyS= tore.java:1274)
247     at = org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:= 1141)
248     at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStor= e.java:1170)
249     at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStor= e.java:1122)
250     at = org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:504)=
251     at = org.apache.cassandra.db.Table.apply(Table.java:441)
252   =   at = org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:3= 38)
253   =   at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)=
254   =   at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)=
255   =   at = java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)<= u>
256   =   at = java.util.concurrent.FutureTask.run(FutureTask.java:138)
257     at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886)
258     at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908)
259     at = java.lang.Thread.run(Thread.java:619)
260  INFO = 11:23:17,640 Finished reading = /var/lib/cassandra/commitlog/CommitLog-1333465745624.log
261 ERROR = 11:23:17,641 Exception encountered during = startup
262 java.lang.RuntimeException: java.util.concurrent.ExecutionException: = java.lang.InternalError: a fault occurred in a recent unsafe memory = access operation in compiled Java code
263     at = org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:508)=
 
Thanks<= /div>
= Michael



= --Apple-Mail=_472F8E16-3115-46FB-8CFF-0649952E1BF4--