incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From srmore <comom...@gmail.com>
Subject Write performance with 1.2.12
Date Fri, 06 Dec 2013 14:10:14 GMT
We have a 3 node cluster running cassandra 1.2.12, they are pretty big
machines 64G ram with 16 cores, cassandra heap is 8G.

The interesting observation is that, when I send traffic to one node its
performance is 2x more than when I send traffic to all the nodes. We ran
1.0.11 on the same box and we observed a slight dip but not half as seen
with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing
CL to ONE make a slight improvement but not much.

The read_Repair_chance is 0.1. We see some compactions running.

following is my iostat -x output, sda is the ssd (for commit log) and sdb
is the spinner.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          66.46    0.00    8.95    0.01    0.00   24.58

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00    27.60  0.00  4.40     0.00   256.00    58.18
0.01    2.55   1.32   0.58
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda2              0.00    27.60  0.00  4.40     0.00   256.00    58.18
0.01    2.55   1.32   0.58
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
dm-0              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
dm-1              0.00     0.00  0.00  0.60     0.00     4.80     8.00
0.00    5.33   2.67   0.16
dm-2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
dm-3              0.00     0.00  0.00 24.80     0.00   198.40     8.00
0.24    9.80   0.13   0.32
dm-4              0.00     0.00  0.00  6.60     0.00    52.80     8.00
0.01    1.36   0.55   0.36
dm-5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
dm-6              0.00     0.00  0.00 24.80     0.00   198.40     8.00
0.29   11.60   0.13   0.32



I can see I am cpu bound here but couldn't figure out exactly what is
causing it, is this caused by GC or Compaction ? I am thinking it is
compaction, I see a lot of context switches and interrupts in my vmstat
output.

I don't see GC activity in the logs but see some compaction activity. Has
anyone seen this ? or know what can be done to free up the CPU.

Thanks,
Sandeep

Mime
View raw message