incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dr. Martin Grabm├╝ller <Martin.Grabmuel...@eleven.de>
Subject RE: Cassandra benchmark shows OK throughput but high read latency (> 100ms)?
Date Tue, 16 Feb 2010 08:32:48 GMT
> The other problem is: if I keep mixed write and read (e.g, 8 
> write threads
> plus 7 read threads) against the 2-nodes cluster 
> continuously, the read
> latency will go up gradually (along with the size of 
> Cassandra data file),
> and at the end it will become ~40ms (up from ~20ms) even with only 15
> threads. During this process the data file grew from 1.6GB to 
> over 3GB even
> if I kept writing the same key/values to Cassandra. It seems 
> that Cassandra
> keeps appending to sstable data files and will only clean up 
> them during
> node cleanup or compact (please correct me if this is incorrect). 

In my tests I have observed that good read latency depends on keeping
the number of data files low.  In my current test setup, I have stored
1.9 TB of data on a single node, which is in 21 data files, and read
latency is between 10 and 60ms (for small reads, larger read of course
take more time).  In earlier stages of my test, I had up to 5000
data files, and read performance was quite bad: my configured 10-second
RPC timeout was regularly encountered.

The number of data files is reduced whenever Cassandra compacts them,
which is either automatically, when enough datafiles are generated by
continuous writing, or when triggered by nodeprobe compact, cleanup etc.

So my advice is to keep the write throughput low enough so that Cassandra
can keep up compacting the data files.  For high write throughput, you need
fast drives, if possible on different RAIDs, which are configured as
different DataDirectories for Cassandra.  On my setup (6 drives in a single
RAID-5 configuration), compaction is quite slow: sequential reads/writes
are done at 150 MB/s, whereas during compaction, read/write-performance
drops to a few MB/s.  You definitively want more than one logical drive,
so that Cassandra can alternate between them when flushin memtables and
when compacting.

I would really be interested whether my observations are shared by other
people on this list.

Thanks!

Martin

Mime
View raw message