cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Freeman, Tim" <tim.free...@hp.com>
Subject Persistently increasing read latency
Date Tue, 01 Dec 2009 18:11:28 GMT
In an 8 hour test run, I've seen the read latency for Cassandra drift fairly linearly from
~460ms to ~900ms.  Eventually my application gets starved for reads and starts misbehaving.
 I have attached graphs -- horizontal scales are seconds, vertical scales are operations per
minute and average milliseconds per operation.  The clearest feature is the light blue line
in the left graph drifting consistently upward during the run.

I have a Cassandra 0.4.1 database, one node, records are 100kbytes each, 350K records, 8 threads
reading, around 700 reads per minute.  There are also 8 threads writing.  This is all happening
on a 4 core processor that's supporting both the Cassandra node and the code that's generating
load for it.  I'm reasonably sure that there are no page faults.

I have attached my storage-conf.xml.  Briefly, it has default values, except RpcTimeoutInMillis
is 30000 and the partitioner is OrderPreservingPartitioner.  Cassandra's garbage collection
parameters are:

   -Xms128m -Xmx1G -XX:SurvivorRatio=8 -XX:+AggressiveOpts -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
 
Is this normal behavior?  Is there some change to the configuration I should make to get it
to stop getting slower?  If it's not normal, what debugging information should I gather? 
Should I give up on Cassandra 0.4.1 and move to a newer version?

I'll leave it running for the time being in case there's something useful to extract from
it.

Tim Freeman
Email: tim.freeman@hp.com
Desk in Palo Alto: (650) 857-2581
Home: (408) 774-1298
Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and Thursday; call my desk
instead.)


Mime
View raw message