Mailing-List: contact cassandra-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-dev@incubator.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Subject: Re: stress insert benchmark & cassandra hangup/crash
From: Eric Evans <eevans@rackspace.com>
To: cassandra-dev@incubator.apache.org
In-Reply-To: <4B15195E.9070204@kreado.com>
References: <4B15195E.9070204@kreado.com>
Content-Type: text/plain; charset="UTF-8"
Date: Tue, 01 Dec 2009 15:24:25 -0600
Message-ID: <1259702665.19220.48.camel@achilles>
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit

On Tue, 2009-12-01 at 15:25 +0200, pavel@kreado.com pop wrote:
> I did simple insert benchmark in java (see code below). It uses 5
> threads to write records in one CF.
> After some point cassandra hangup, then crashes. I use single node
> cassandra installation with commit log on separate hdd.
> 
> Hardware:
> AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
> 4 GB RAM,
> 2xHDD Samsung 1TB 32MB cache 7200RPM

[ ... ]

> After long pause cassandra crashes (i use 1GB for java heap):
> 
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid21132.hprof ...
> Heap dump file created [1196864899 bytes in 51.159 secs]
> 
> I tried this testcase on three mashines with latest development
> version
> of cassandra and with 0.4.2. And i got the same problems.

[ ... ]

>           client.insert("suggy",
>                         String.valueOf(i),
>                         cp,
>                        
> "================================================================================".getBytes("UTF-8"),
>                         timestamp,
>                         ConsistencyLevel.ZERO);

With ConsistencyLevel.ZERO writes are completely asynchronous, so one
possibility is that you are simply pushing more writes than the node can
handle and it's getting further and further behind, eventually
exhausting the heap.

You should probably also read
http://wiki.apache.org/cassandra/MemtableThresholds if you haven't
already.

-- 
Eric Evans
eevans@rackspace.com