incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gurpreet Singh <gurpreet.si...@gmail.com>
Subject cassandra read latency help
Date Thu, 17 May 2012 02:33:47 GMT
Hi,

Requirements:

I am looking to setup a cluster of cassandra 1.0.9.
The cluster nodes will have just 1 keyspace with 1 column family.
RF will be 2 per datacenter and number of datacenters is 2.
The key size is 8 bytes, the rows are expected to have 2 columns and total
about 400 bytes in size.
The number of keys are expected to be 500 million total, while the hot data
size would probably be around 1 million or so.
We want the cluster to support a qps of 100 reads per second with latency
per read < 30 ms.
Read batch size would be about 20 keys coming in as a multiget slice query.
Write qps would be 50 writes per second
read/write consistency level would be 1.


hardware:
Each node has 8 cpus, 16 gb RAM, 1 boot drive, commitlog drive mirrored, 2
data drives 250 gb each striped with stripe size 512 kb

I am trying to do an experiment with a 1-node cluster by running a load
test on this and able to configure for maximum performance.
While with multiple threads i am able to achieve the qps rates even on this
1 node, the latency numbers as seen in cfstats are high, almost 60 ms.

Following is the configuration of the node:

heap size = 4 gigs
     memtable_total_space_in_mb = 2 gig
     key cache size (1 million) (shoudl take about 1 gig in RAM)

row cache (off heap) - 1 million (should take about 4-5 gigs in RAM)

I inserted about 120 million keys in this cluster. The write qps is awesome.
I am running a test now with 6 threads doing reads, the whole system
throttled at 25 reads per second. While i am able to achieve this easily,
the latency times per read are high.

cfstats:
number of keys: 120 million
number of sstables: 9
read latency - 55 ms  (this is the number i am looking to bring down)
row cache hit ratio is 60%
key cache hit ratio is 7%

iostats:
r/s = 78
r kb/s = 366
avgrq-sz = 8.37
avgqu-sz = 2.6
await = 45 ms
svctm = 3 ms
%util = 15%

Any ideas on what could help here bring down the read latency even more ?
Thanks
Gurpreet

Mime
View raw message