incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe <watche...@gmail.com>
Subject Read performance vs. vmstat + your experience with read optimizations
Date Mon, 20 Jun 2011 19:39:57 GMT
Hi all,
I am having trouble reconciling various metrics regarding reads so I'm
hoping someone here can help me understand what's going on.

I am running tests on a single node cluster with 16GB of RAM. I'm testing on
the following column family:
               Column Family: PUBLIC_MONTHLY
                SSTable count: 1
                Space used (live): 28468417160
                Space used (total): 28468417160
                Memtable Columns Count: 0
                Memtable Data Size: 0
                Memtable Switch Count: 0
                Read Count: 2669019991
                Read Latency: 0.846 ms.
                Write Count: 0
                Write Latency: NaN ms.
                Pending Tasks: 0
                Key cache capacity: 20000
                Key cache size: 20000
                Key cache hit rate: 0.33393368358762754
                Row cache capacity: 50000
                Row cache size: 50000
                Row cache hit rate: 0.15195090894076155
                Compacted row minimum size: 216
                Compacted row maximum size: 88148
                Compacted row mean size: 483
The keys represent a grid cells (65 million), columns to store monthly
increments (total & sum, to produce averages), super columns tag the data
source
The mean row length is 483 bytes

The keycache & rowcache enabled but kept very low just to test going through
the disk since I expect very random reads in production.

I've done everything I can to optimize reads
 - Cassandra is setup to use only 4GB because my dataset is 28GB
 - I've compacted the data to a single file
 - I'm hitting cassandra with only 1 read request at a time & no writes. The
request is a multislice across hundreds or thousands of keys

The problem:
vmstat shows that Cassandra is doing about 200MB/s of IO and since there are
no writes on the system, I know it can only be reading (RAID-0 SSD drives).
I know that Cassandra is reading about 1/3 the super columns. To be safe,
let's assume Cassandra is deserializing 1/2 the row.
I'll just assume for simplicity that the row size is 512 bytes.

So it looks to me as if Cassandra is deserializing
200MB/((512bytes)/2)=400MB/(0.5KB) = 800K rows per second.
That's 800 keys per millisecond.

And yet, my app is being throttled by Cassandra during its
MultigetSuperSliceCounterQuery: measuring the time spent in Hector show that
I'm getting at most 20-30 rows per ms and sometimes I get

My questions:
1) Any idea where the discrepency can come from ?
I'd like to believe there is some magic setting that will x10 my read
performance...

2) How do you recommend allocating memory ? Should I give the OS cache as
much as possible or should I max out Cassandra's cache ?

3) Does anyone have numbers regarding the performance of range queries when
compared to multiget queries ? I can probably take SimpleGeo's idea of a
Z-order code to map the 2D grid to 1D ranges but I wonder if I will get the
x10 performance I'm looking for.

PS:Nodetool indicates that the read latency is 0.846ms so that's 1.12 key/ms
?! Let's just leave this aside, the process hasbeen running for 12 hours and
maybe the number are very different from what we're seeing here.

Thanks
PG

vmstat (SSD not maxed out in this but it does at other times)
 0  0  78184  89252  10764 11254784    0    0 186448    18 8002 2352  7  4
50 39
 0  9  78184  88880  10764 11249900    0    0 176602    78 8046 2957  7  3
64 26
 0 16  78184  88260  10764 11246824    0    0 195726     0 9090 2718  8  4
52 36
 0 14  78184  89376  10764 11242496    0    0 227858     0 9533 2444  7  4
45 44
 0  0  78184  88260  10764 11254336    0    0 203374     1 9144 2567  7  4
59 30
 0  4  78184  90368  10764 11251856    0    0 235394     0 9732 1827  6  4
52 38
 0 23  78184  92352  10756 11238000    0    0 203140    98 9007 2835  7  4
59 29
 0  0  78184  91608  10756 11250952    0    0 176348     0 8354 3535  7  3
64 26
 1  0  78184  92352  10756 11250228    0    0 163952     0 7475 3243  9  3
57 31

iostat -dmx 2 (filtered)
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sdb              80.00     0.00 4061.50    0.00    94.34     0.00    47.57
 80.18   19.49   19.49    0.00   0.16  63.00
sda              78.50     0.00 3934.50    0.00    94.72     0.00    49.31
 76.87   19.27   19.27    0.00   0.16  62.80
dm-0              0.00     0.00 8310.50    0.00   192.47     0.00    47.43
169.89   20.15   20.15    0.00   0.08  63.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sdb             101.50     0.00 5141.00    0.00   121.16     0.00    48.27
103.29   20.03   20.03    0.00   0.16  80.60
sda             100.00     0.00 5190.50    0.00   121.59     0.00    47.97
100.74   19.24   19.24    0.00   0.15  79.80
dm-0              0.00     0.00 10552.50    0.00   242.85     0.00    47.13
  219.09   20.57   20.57    0.00   0.08  81.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sdb              67.50     0.00 3692.50    0.00    86.23     0.00    47.83
 64.89   17.92   17.92    0.00   0.15  57.00
sda              90.00     0.00 3680.00    0.00    87.22     0.00    48.54
 70.86   19.77   19.77    0.00   0.16  57.40
dm-0              0.00     0.00 7364.00    0.00   170.29     0.00    47.36
145.79   20.39   20.39    0.00   0.08  58.20

iming examples from my app
numRollupKeys=13312,getdata_ms=617 => 21.57 keys/ms
numRollupKeys=6144,getdata_ms=224  => 27.42
numRollupKeys=14080,getdata_ms=793 => 17.75
numRollupKeys=8448,getdata_ms=157  => 53.08
numRollupKeys=6400,getdata_ms=601  => 10.64
numRollupKeys=7680,getdata_ms=550  => 13.96
numRollupKeys=12800,getdata_ms=720 => 17.77
numRollupKeys=6912,getdata_ms=275  => 25.14

Mime
View raw message