Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 74.125.82.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=STHSV1zok2VhHlrDXWVtrs9rYnSmSeZinJ5PUBRDG96pBHTZ+WrZ8m+W+P6+0POCD9
         ZnGHApxFiCCZ0Hl6zaFglT+CMpAS9mlcidjx80Xvh3QIVQZPLVyPDUIe9NHshcD0W+hr
         get8I7BXmUPoUz5usfquuFTbxhP0GXo2cnRVM=
MIME-Version: 1.0
In-Reply-To: <AANLkTikakT-n3KA1g_lNjczJDrLdRzhrSZ2oMAOXXF8t@mail.gmail.com>
References: <AANLkTikakT-n3KA1g_lNjczJDrLdRzhrSZ2oMAOXXF8t@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Thu, 27 May 2010 08:38:52 -0600
Message-ID: <AANLkTilhynnDqOs0rXnXaz9gmITa_36YDHLYGzqePm6E@mail.gmail.com>
Subject: Re: Cassandra-0.6.1 Crash Error: out of memory
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

It looks like you simply don't have a large enough heap for all the
in-flight data.

Low-hanging fruit includes

 - upgrade to 0.6.2 (available from http://people.apache.org/~eevans/
until release is official later today)
 - when you get a TimeoutException on the client, sleep 100ms or so
before retrying
 - increase the max heap size

On Tue, May 25, 2010 at 9:10 PM, Peng Guo <gpcuster@gmail.com> wrote:
>
> Hi
>
> There are 3 Cassandra=A0 servcer runing, and 18 process insert lots of da=
ta
> into the=A0 Cassandra=A0 servcer.
> After runing for an hour=A0 the Cassandra=A0 servcer crashed.
> The error msg is below:
>
> INFO [GC inspection] 2010-05-26 00:56:50,153 GCInspector.java (line 110) =
GC
> for ConcurrentMarkSweep: 7764 ms, 120920 reclaimed leaving 2168941512 use=
d;
> max is 2284584960
> =A0INFO [GC inspection] 2010-05-26 01:01:51,803 GCInspector.java (line 11=
0) GC
> for ConcurrentMarkSweep: 5368 ms, 214864 reclaimed leaving 2168850488 use=
d;
> max is 2284584960
> ERROR [pool-1-thread-63] 2010-05-26 01:10:34,721 Cassandra.java (line 161=
8)
> Internal error processing batch_mutate
> java.lang.OutOfMemoryError: Java heap space
> ERROR [pool-1-thread-55] 2010-05-26 01:10:29,157 CassandraDaemon.java (li=
ne
> 78) Fatal exception in thread Thread[pool-1-thread-55,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [main] 2010-05-26 01:10:29,157 CassandraDaemon.java (line 195)
> Exception encountered during startup.
> java.lang.OutOfMemoryError: Java heap space
> ERROR [CACHETABLE-TIMER-2] 2010-05-26 01:04:58,068 CassandraDaemon.java
> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-2,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [pool-1-thread-56] 2010-05-26 01:14:35,482 CassandraDaemon.java (li=
ne
> 78) Fatal exception in thread Thread[pool-1-thread-56,5,main]
> java.lang.OutOfMemoryError: Java heap space
> =A0INFO [GC inspection] 2010-05-26 01:14:35,482 GCInspector.java (line 11=
0) GC
> for ConcurrentMarkSweep: 7702 ms, 15440656 reclaimed leaving 2153622464
> used; max is 2284584960
> ERROR [HINTED-HANDOFF-POOL:1] 2010-05-26 01:14:35,274
> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
> =A0=A0=A0=A0=A0=A0=A0 at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.FutureTask.get(FutureTask.j=
ava:83)
> =A0=A0=A0=A0=A0=A0=A0 at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute=
(DebuggableThreadPoolExecutor.java:86)
> =A0=A0=A0=A0=A0=A0=A0 at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor=
.java:888)
> =A0=A0=A0=A0=A0=A0=A0 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav=
a:908)
> =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> ERROR [pool-1-thread-50] 2010-05-26 01:12:58,865 CassandraDaemon.java (li=
ne
> 78) Fatal exception in thread Thread[pool-1-thread-50,5,main]
> java.lang.OutOfMemoryError: Java heap space
>
> --
> Regards
> =A0 =A0Peng Guo
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com