From user-return-25350-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu Apr 5 04:31:39 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ADD5D9263 for ; Thu, 5 Apr 2012 04:31:39 +0000 (UTC) Received: (qmail 95484 invoked by uid 500); 5 Apr 2012 04:31:37 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 95422 invoked by uid 500); 5 Apr 2012 04:31:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 95394 invoked by uid 99); 5 Apr 2012 04:31:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Apr 2012 04:31:34 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [74.125.82.44] (HELO mail-wg0-f44.google.com) (74.125.82.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Apr 2012 04:31:29 +0000 Received: by wgbdr13 with SMTP id dr13so738165wgb.25 for ; Wed, 04 Apr 2012 21:31:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=+IN8N0nqDUyNp/aWtNV7dtwqdqqkEmHh8avFsY/91aI=; b=dqMa7/GsvNxpO2H0JqfkrR/Tk1J0+ioUYiEmKnmo8Gh0jfeju/KW81YMJpKG3xDZko THvOE33P75+Nd5jjov9418c0wlZPhLT8C+wH+tdA5qTjZrQQeUIBMdsgH7vIZbQGPV8J UCp9pcB3gYtUqFviJhwJcQzF2dBgbGJLizU9ZS67v7Wv+virBQJqoTZY2WDutysZcr9y tKyNcobDgar6kzSOEA1+plRRiP167VnP6Auu+lv57exsihdL4exMooyausv6NXo7FIVt p49w359v6fHPEjvVogarUFahNNktIbWSLbqK7tpQt9zUf5K8BrlZgha7EUGXHkvFdmxs nhVw== MIME-Version: 1.0 Received: by 10.216.134.155 with SMTP id s27mr766632wei.80.1333600267670; Wed, 04 Apr 2012 21:31:07 -0700 (PDT) Received: by 10.180.80.73 with HTTP; Wed, 4 Apr 2012 21:31:07 -0700 (PDT) In-Reply-To: <49AA9395-02A8-4EE5-93D3-549196C2FF16@thelastpickle.com> References: <003001cd1256$c9423820$5bc6a860$@com> <49AA9395-02A8-4EE5-93D3-549196C2FF16@thelastpickle.com> Date: Thu, 5 Apr 2012 07:31:07 +0300 Message-ID: Subject: Re: server down From: Michael Vaknin To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001636d34cd23a133504bce702c5 X-Gm-Message-State: ALoCoQnlL3GuEk6iAtceJES4c2bPQEHrfMdNPJhlp/sY0MXk5aacIYcMa53dqB9btROJkmuZ7w43 X-Virus-Checked: Checked by ClamAV on apache.org --001636d34cd23a133504bce702c5 Content-Type: text/plain; charset=ISO-8859-1 i am usuinc cassandra 1.0.3 java 6.0.17 on ubuntu 10.04 this is a stable version for 6 months now. On Wed, Apr 4, 2012 at 9:51 PM, aaron morton wrote: > What version of cassandra are you using ? > > What java vendor / version ? > > What OS vendor / version ? > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 4/04/2012, at 11:33 PM, Michael Vaknine wrote: > > I am recently having problems with 1 of my 4ty cluster servers.**** > I would appreciate any help**** > ** ** > ERROR 11:23:14,331 Fatal exception in thread > Thread[MutationStage:19,5,main]**** > 225 java.lang.InternalError: a fault occurred in a recent unsafe memory > access operation in compiled Java code**** > 226 at > org.apache.cassandra.io.util.AbstractDataInput.readInt(AbstractDataInput.java:196) > **** > 227 at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:348) > **** > 228 at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120) > **** > 229 at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83) > **** > 230 at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73) > **** > 231 at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37) > **** > 232 at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79) > **** > 233 at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40) > **** > 234 at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > **** > 235 at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > **** > 236 at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107) > **** > 237 at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145) > **** > 238 at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124) > **** > 239 at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98) > **** > 240 at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > **** > 241 at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > **** > 242 at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116) > **** > 243 at > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:144) > **** > 244 at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:227) > **** > 245 at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62) > **** > 246 at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1274) > **** > 247 at > org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1141) > **** > 248 at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1170) > **** > 249 at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1122) > **** > 250 at > org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:504)*** > * > 251 at org.apache.cassandra.db.Table.apply(Table.java:441)**** > 252 at > org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:338) > **** > 253 at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)*** > * > 254 at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)*** > * > 255 at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)**** > 256 at java.util.concurrent.FutureTask.run(FutureTask.java:138)**** > 257 at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > **** > 258 at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > **** > 259 at java.lang.Thread.run(Thread.java:619)**** > 260 INFO 11:23:17,640 Finished reading > /var/lib/cassandra/commitlog/CommitLog-1333465745624.log**** > 261 ERROR 11:23:17,641 Exception encountered during startup**** > 262 java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.InternalError: a fault occurred in a recent unsafe memory access > operation in compiled Java code**** > 263 at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:508) > **** > ** ** > Thanks**** > Michael**** > > > --001636d34cd23a133504bce702c5 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
i am usuinc cassandra 1.0.3 java 6.0.17 on ubuntu 10.04this is a stable version for 6 months now.

On Wed, Apr 4, 2012 at 9:51 PM, aaron morton <aaron@thelastpickle.com> wrote:
What ver= sion of cassandra are you using ?=A0

What java vendor / = version ?=A0

What OS vendor / version ?=A0

= Cheers

<= div style=3D"word-wrap:break-word">
-----------------
Aaron Morton
Freelance Deve= loper
@aaronmorton

On 4/04/2012, at 11:33 PM, Michael Vaknine wrote:

I am recently= having problems with 1 of my 4ty cluster servers.
I would appreciate any help
=A0
ERROR 11:23:14,331 Fatal exception in thread Thread[MutationStage:19,5,main= ]
225 java.lang.InternalError: a fault occurred in a recent unsafe memory acc= ess operation in compiled Java code
226=A0=A0=A0=A0 at org.apache.cassandra.io.util.AbstractDataInput.readInt(A= bstractDataInput.java:196)
227=A0=A0=A0=A0 at org.apache.cassandra.utils.ByteBufferUtil.readWithLength= (ByteBufferUtil.java:348)
228=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnSerializer.deserialize(Col= umnSerializer.java:120)
229=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnSerializer.deserialize(Col= umnSerializer.java:83)
230=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnSerializer.deserialize(Col= umnSerializer.java:73)
231=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnSerializer.deserialize(Col= umnSerializer.java:37)
232=A0=A0=A0=A0 at org.apache.cassandra.db.columniterator.SimpleSliceReader= .computeNext(SimpleSliceReader.java:79)
233=A0=A0=A0=A0 at org.apache.cassandra.db.columniterator.SimpleSliceReader= .computeNext(SimpleSliceReader.java:40)
234=A0=A0=A0=A0 at com.google.common.collect.AbstractIterator.tryToComputeN= ext(AbstractIterator.java:140)
235=A0=A0=A0=A0 at com.google.common.collect.AbstractIterator.hasNext(Abstr= actIterator.java:135)
236=A0=A0=A0=A0 at org.apache.cassandra.db.columniterator.SSTableSliceItera= tor.hasNext(SSTableSliceIterator.java:107)
237=A0=A0=A0=A0 at org.apache.cassandra.utils.MergeIterator$Candidate.advan= ce(MergeIterator.java:145)
238=A0=A0=A0=A0 at org.apache.cassandra.utils.MergeIterator$ManyToOne.advan= ce(MergeIterator.java:124)
239=A0=A0=A0=A0 at org.apache.cassandra.utils.MergeIterator$ManyToOne.compu= teNext(MergeIterator.java:98)
240=A0=A0=A0=A0 at com.google.common.collect.AbstractIterator.tryToComputeN= ext(AbstractIterator.java:140)
241=A0=A0=A0=A0 at com.google.common.collect.AbstractIterator.hasNext(Abstr= actIterator.java:135)
242=A0=A0=A0=A0 at org.apache.cassandra.db.filter.SliceQueryFilter.collectR= educedColumns(SliceQueryFilter.java:116)
243=A0=A0=A0=A0 at org.apache.cassandra.db.filter.QueryFilter.collateColumn= s(QueryFilter.java:144)
244=A0=A0=A0=A0 at org.apache.cassandra.db.CollationController.collectAllDa= ta(CollationController.java:227)
245=A0=A0=A0=A0 at org.apache.cassandra.db.CollationController.getTopLevelC= olumns(CollationController.java:62)
246=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelCol= umns(ColumnFamilyStore.java:1274)
247=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(Colum= nFamilyStore.java:1141)
248=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamil= y(ColumnFamilyStore.java:1170)
249=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamil= y(ColumnFamilyStore.java:1122)
250=A0=A0=A0=A0 at org.apache.cassandra.db.Table.readCurrentIndexedColumns(= Table.java:504)
251=A0=A0=A0=A0 at org.apache.cassandra.db.Table.apply(Table.java:441)
252= =A0=A0=A0=A0 at org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(C= ommitLog.java:338)
253=A0=A0=A0=A0 at= org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)=
254=A0=A0=A0=A0 at= java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)=
255=A0=A0=A0=A0 at= java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)<= u>
256=A0=A0=A0=A0 at= java.util.concurrent.FutureTask.run(FutureTask.java:138)
257=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(T= hreadPoolExecutor.java:886)
258=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa= dPoolExecutor.java:908)
259 =A0=A0=A0=A0at java.lang.Thread.run(Thread.java:619)
260=A0 INFO 11:23= :17,640 Finished reading /var/lib/cassandra/commitlog/CommitLog-13334657456= 24.log
261 ERROR 11:23:17= ,641 Exception encountered during startup
262 java.lang.RuntimeException: java.util.concurrent.ExecutionException: ja= va.lang.InternalError: a fault occurred in a recent unsafe memory access op= eration in compiled Java code
263=A0=A0=A0=A0 at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBU= tilities.java:508)
=A0
Thanks
Michael

=
--001636d34cd23a133504bce702c5--