Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 209.85.219.220 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ZReSWUceUqclgh0Nul8T0s7UWsGOaDXzOQAYfnXQGiRa78NpKlRw9T8dhULVhTr3ur
         y6K491HH4zB5FuIaQP7ATvo9T2cVxuI/svA0jCzjO29D/w8HUPsFW0Ar81ekMdJu3+S0
         874JYvMdp3FdqM9bTZ9l1sLmqkeRHX+iI2wUI=
MIME-Version: 1.0
In-Reply-To: 
 <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850190C6@GVW0432EXB.americas.hpqcorp.net>
References: 
 <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850190C6@GVW0432EXB.americas.hpqcorp.net>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Tue, 1 Dec 2009 13:10:00 -0600
Message-ID: <e06563880912011110y354e1338mc8a255934e8263f4@mail.gmail.com>
Subject: Re: Persistently increasing read latency
To: cassandra-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

1) use jconsole to see what is happening to jvm / cassandra internals.
 possibly you are slowly exceeding cassandra's ability to keep up with
writes, causing the jvm to spend more and more effort GCing to find
enough memory to keep going

2) you should be at least on 0.4.2 and preferably trunk if you are
stress testing

-Jonathan

On Tue, Dec 1, 2009 at 12:11 PM, Freeman, Tim <tim.freeman@hp.com> wrote:
> In an 8 hour test run, I've seen the read latency for Cassandra drift fai=
rly linearly from ~460ms to ~900ms. =A0Eventually my application gets starv=
ed for reads and starts misbehaving. =A0I have attached graphs -- horizonta=
l scales are seconds, vertical scales are operations per minute and average=
 milliseconds per operation. =A0The clearest feature is the light blue line=
 in the left graph drifting consistently upward during the run.
>
> I have a Cassandra 0.4.1 database, one node, records are 100kbytes each, =
350K records, 8 threads reading, around 700 reads per minute. =A0There are =
also 8 threads writing. =A0This is all happening on a 4 core processor that=
's supporting both the Cassandra node and the code that's generating load f=
or it. =A0I'm reasonably sure that there are no page faults.
>
> I have attached my storage-conf.xml. =A0Briefly, it has default values, e=
xcept RpcTimeoutInMillis is 30000 and the partitioner is OrderPreservingPar=
titioner. =A0Cassandra's garbage collection parameters are:
>
> =A0 -Xms128m -Xmx1G -XX:SurvivorRatio=3D8 -XX:+AggressiveOpts -XX:+UsePar=
NewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>
> Is this normal behavior? =A0Is there some change to the configuration I s=
hould make to get it to stop getting slower? =A0If it's not normal, what de=
bugging information should I gather? =A0Should I give up on Cassandra 0.4.1=
 and move to a newer version?
>
> I'll leave it running for the time being in case there's something useful=
 to extract from it.
>
> Tim Freeman
> Email: tim.freeman@hp.com
> Desk in Palo Alto: (650) 857-2581
> Home: (408) 774-1298
> Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and Th=
ursday; call my desk instead.)
>
>