cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: cassandra read latency help
Date Wed, 30 May 2012 23:38:02 GMT
> 80 ms per request
sounds high. 

I'm doing some guessing here, i am guessing memory usage is the problem..

* I assume you are not longer seeing excessive GC activity. 
* The key cache will not get used when you hit the row cache. I would disable the row cache
if you have a random workload, which it looks like you do. 
* 500 million is a lot of keys to have on a single node. At the default index sample of every
128 keys it will have about 4 million samples, which is probably taking up a lot of memory.


Is this testing a real world scenario or an abstract benchmark ? IMHO you will get more insight
from testing something that resembles your application. 

Cheers
 
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 26/05/2012, at 8:48 PM, Gurpreet Singh wrote:

> Hi Aaron,
> Here is the latest on this..
> i switched to a node with 6 disks and running some read tests, and i am seeing something
weird.
> 
> setup:
> 1 node, cassandra 1.0.9, 8 cpu, 16 gig RAM, 6 7200 rpm SATA data disks striped 512 kb,
commitlog mirrored.
> 1 keyspace with just 1 column family
> random partitioner
> total number of keys: 500 million (the keys are just longs from 1 to 500 million) 
> avg key size: 8 bytes
> bloom filter size: 1 gig
> total disk usage: 70 gigs compacted 1 sstable
> mean compacted row size: 149 bytes
> heap size: 8 gigs
> keycache size: 2 million (takes around 2 gigs in RAM)
> rowcache size: 1 million (off-heap)
> memtable_total_space_mb : 2 gigs
> 
> test: 
> Trying to do 5 reads per second. Each read is a multigetslice query for just 1 key, 2
columns. 
> 
> observations:
> row cache hit rate: 0.4
> key cache hit rate: 0.0 (this will increase later on as system moves to steady state)
> cfstats - 80 ms
> 
> iostat (every 5 seconds): 
> 
> r/s : 400   
> %util: 20%  (all disks are at equal utilization)
> await: 65-70 ms (for each disk)
> svctm : 2.11 ms (for each disk)
> r-kB/s - 35000
> 
> why this is weird is because.. 
> 5 reads per second is causing a latency of 80 ms per request (according to cfstats).
isnt this too high?
> 35 MB/s is being read from the disk. That is again very weird. This number is way too
high, avg row size is just 149 bytes. Even index reads should not cause this high data being
read from the disk.
> 
> what i understand is that each read request translates to 2 disk accesses (because there
is only 1 sstable). 1 for the index, 1 for the data. At such a low reads/second, why is the
latency so high?
> 
> would appreciate help debugging this issue.
> Thanks
> Gurpreet
> 
> 
> On Tue, May 22, 2012 at 2:46 AM, aaron morton <aaron@thelastpickle.com> wrote:
> With
> 
>> heap size = 4 gigs
> 
> I would check for GC activity in the logs and consider setting it to 8 given you have
16 GB.  You can also check if the IO system is saturated (http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html)
Also take a look at nodetool cfhistogram perhaps to see how many sstables are involved. 
> 
> 
> I would start by looking at the latency reported on the server, then work back to the
client….
> 
> I may have missed it in the email but what recent latency for the CF is reported by nodetool
cfstats ? That's latency for a single request on a single read thread. The default settings
give you 32 read threads. 
> 
> If you know the latency for a single request, and you know you have 32 concurrent read
threads, you can get an idea of the max throughput for a single node. Once you get above that
throughput the latency for a request will start to include wait time. 
> 
> It's a bit more complicated, because when you request 40 rows that turns into 40 read
tasks. So if two clients send a request for 40 rows at the same time there will be 80 read
tasks to be processed by 32 threads. 
> 
> Hope that helps. 
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 20/05/2012, at 4:10 PM, Radim Kolar wrote:
> 
>> Dne 19.5.2012 0:09, Gurpreet Singh napsal(a):
>>> Thanks Radim.
>>> Radim, actually 100 reads per second is achievable even with 2 disks.
>> it will become worse as rows will get fragmented.
>>> But achieving them with a really low avg latency per key is the issue.
>>> 
>>> I am wondering if anyone has played with index_interval, and how much of a difference
would it make to reads on reducing the index_interval.
>> close to zero. but try it yourself too and post your findings.
> 
> 


Mime
View raw message