Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates
 74.125.83.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=q7knBf+jqSaH4dxrFn3jQ7ETdojP8xmMs8d/Hm9whtB5HWdvqVojBs4QC1Rzi1gMvG
         lCLhXJYpXcM6NEW3YteKFSyItwXtWdi+GV2hAXOG5juDyzXUsiOy5dNLVCrO2UlXntk6
         cYUcVi/si5uoAE4q4nanv8TI2UNpGpUsgS0RE=
MIME-Version: 1.0
In-Reply-To: <43B93B23-D289-481A-8C69-8BCB866ADFF9@toptarif.de>
References: <AANLkTik0uC_vX=8obMdnVcetia15qQa6AtnHoaaaq2yw@mail.gmail.com>
	<AANLkTi=u5gNFhc9mmm8mD3K0d7oqGY74wAmV5nE4PhcT@mail.gmail.com>
	<43B93B23-D289-481A-8C69-8BCB866ADFF9@toptarif.de>
Date: Sun, 12 Dec 2010 10:21:50 -0600
Message-ID: <AANLkTim=56YpuvmF=N8LK_fsZMwVk3O-j05E65=FsjNN@mail.gmail.com>
Subject: Re: Memory leak with Sun Java 1.6 ?
From: Jonathan Ellis <jbellis@gmail.com>
To: user <user@cassandra.apache.org>
Content-Type: multipart/alternative; boundary=001636e0b34c186ba1049738fcba

--001636e0b34c186ba1049738fcba
Content-Type: text/plain; charset=ISO-8859-1

http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors

On Sun, Dec 12, 2010 at 9:52 AM, Timo Nentwig <timo.nentwig@toptarif.de>wrote:

>
> On Dec 10, 2010, at 19:37, Peter Schuller wrote:
>
> > To cargo cult it: Are you running a modern JVM? (Not e.g. openjdk b17
> > in lenny or some such.) If it is a JVM issue, ensuring you're using a
> > reasonably recent JVM is probably much easier than to start tracking
> > it down...
>
> I had OOM problems with OpenJDK, switched to Sun/Oracle's recent 1.6.0_23
> and...still have the same problem :-\ Stack trace always looks the same:
>
> java.lang.OutOfMemoryError: Java heap space
>        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>        at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)
>        at
> org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:261)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76)
>        at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>        at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>        at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
>        at
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:383)
>        at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:393)
>        at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:351)
>        at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:52)
>        at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
>
> I'm writing from 1 client with 50 threads to a cluster of 4 machines (with
> hector). With QUORUM and ONE 2 machines quite reliably will soon die with
> OOM. What may cause this? Won't cassandra block/reject when memtable is full
> and being flushed to disk but grow and if flushing to disk isn't fast enough
> will run out of memory?


-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

--001636e0b34c186ba1049738fcba
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<a href=3D"http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-=
dying-with-oom-errors">http://www.riptano.com/docs/0.6/troubleshooting/inde=
x#nodes-are-dying-with-oom-errors</a><br><br><div class=3D"gmail_quote">On =
Sun, Dec 12, 2010 at 9:52 AM, Timo Nentwig <span dir=3D"ltr">&lt;<a href=3D=
"mailto:timo.nentwig@toptarif.de">timo.nentwig@toptarif.de</a>&gt;</span> w=
rote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; borde=
r-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class=3D"im"=
><br>
On Dec 10, 2010, at 19:37, Peter Schuller wrote:<br>
<br>
&gt; To cargo cult it: Are you running a modern JVM? (Not e.g. openjdk b17<=
br>
&gt; in lenny or some such.) If it is a JVM issue, ensuring you&#39;re usin=
g a<br>
&gt; reasonably recent JVM is probably much easier than to start tracking<b=
r>
&gt; it down...<br>
<br>
</div>I had OOM problems with OpenJDK, switched to Sun/Oracle&#39;s recent =
1.6.0_23 and...still have the same problem :-\ Stack trace always looks the=
 same:<br>
<br>
java.lang.OutOfMemoryError: Java heap space<br>
 =A0 =A0 =A0 =A0at java.nio.HeapByteBuffer.&lt;init&gt;(HeapByteBuffer.java=
:57)<br>
 =A0 =A0 =A0 =A0at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.utils.FBUtilities.readByteArray(FBU=
tilities.java:261)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnSerializer.deserialize(Col=
umnSerializer.java:76)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnSerializer.deserialize(Col=
umnSerializer.java:35)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilySerializer.deseriali=
zeColumns(ColumnFamilySerializer.java:129)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilySerializer.deseriali=
ze(ColumnFamilySerializer.java:120)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationSerializer.defreezeTh=
eMaps(RowMutation.java:383)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationSerializer.deserializ=
e(RowMutation.java:393)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationSerializer.deserializ=
e(RowMutation.java:351)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(Ro=
wMutationVerbHandler.java:52)<br>
 =A0 =A0 =A0 =A0at org.apache.cassandra.net.MessageDeliveryTask.run(Message=
DeliveryTask.java:63)<br>
 =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor.runWorker(Thread=
PoolExecutor.java:1110)<br>
 =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa=
dPoolExecutor.java:603)<br>
 =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636)<br>
<br>
I&#39;m writing from 1 client with 50 threads to a cluster of 4 machines (w=
ith hector). With QUORUM and ONE 2 machines quite reliably will soon die wi=
th OOM. What may cause this? Won&#39;t cassandra block/reject when memtable=
 is full and being flushed to disk but grow and if flushing to disk isn&#39=
;t fast enough will run out of memory?</blockquote>
</div><br><br clear=3D"all"><br>-- <br>Jonathan Ellis<br>Project Chair, Apa=
che Cassandra<br>co-founder of Riptano, the source for professional Cassand=
ra support<br><a href=3D"http://riptano.com">http://riptano.com</a><br>

--001636e0b34c186ba1049738fcba--