incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vicky Kak <vicky....@gmail.com>
Subject Re: Write performance with 1.2.12
Date Fri, 06 Dec 2013 15:32:48 GMT
Hard to say much without knowing about the cassandra configurations.
Yes compactions/GC's could skipe the CPU, I had similar behavior with my
setup.

-VK


On Fri, Dec 6, 2013 at 7:40 PM, srmore <comomore@gmail.com> wrote:

> We have a 3 node cluster running cassandra 1.2.12, they are pretty big
> machines 64G ram with 16 cores, cassandra heap is 8G.
>
> The interesting observation is that, when I send traffic to one node its
> performance is 2x more than when I send traffic to all the nodes. We ran
> 1.0.11 on the same box and we observed a slight dip but not half as seen
> with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing
> CL to ONE make a slight improvement but not much.
>
> The read_Repair_chance is 0.1. We see some compactions running.
>
> following is my iostat -x output, sda is the ssd (for commit log) and sdb
> is the spinner.
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           66.46    0.00    8.95    0.01    0.00   24.58
>
> Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sda               0.00    27.60  0.00  4.40     0.00   256.00    58.18
> 0.01    2.55   1.32   0.58
> sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 0.00    0.00   0.00   0.00
> sda2              0.00    27.60  0.00  4.40     0.00   256.00    58.18
> 0.01    2.55   1.32   0.58
> sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 0.00    0.00   0.00   0.00
> sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 0.00    0.00   0.00   0.00
> dm-0              0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 0.00    0.00   0.00   0.00
> dm-1              0.00     0.00  0.00  0.60     0.00     4.80     8.00
> 0.00    5.33   2.67   0.16
> dm-2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 0.00    0.00   0.00   0.00
> dm-3              0.00     0.00  0.00 24.80     0.00   198.40     8.00
> 0.24    9.80   0.13   0.32
> dm-4              0.00     0.00  0.00  6.60     0.00    52.80     8.00
> 0.01    1.36   0.55   0.36
> dm-5              0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 0.00    0.00   0.00   0.00
> dm-6              0.00     0.00  0.00 24.80     0.00   198.40     8.00
> 0.29   11.60   0.13   0.32
>
>
>
> I can see I am cpu bound here but couldn't figure out exactly what is
> causing it, is this caused by GC or Compaction ? I am thinking it is
> compaction, I see a lot of context switches and interrupts in my vmstat
> output.
>
> I don't see GC activity in the logs but see some compaction activity. Has
> anyone seen this ? or know what can be done to free up the CPU.
>
> Thanks,
> Sandeep
>
>
>

Mime
View raw message