incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From srmore <comom...@gmail.com>
Subject Re: Write performance with 1.2.12
Date Fri, 06 Dec 2013 16:36:04 GMT
On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak <vicky.kak@gmail.com> wrote:

> You have passed the JVM configurations and not the cassandra
> configurations which is in cassandra.yaml.
>

Apologies, was tuning JVM and that's what was in my mind.
Here are the cassandra settings http://pastebin.com/uN42GgYT



> The spikes are not that significant in our case and we are running the
> cluster with 1.7 gb heap.
>
> Are these spikes causing any issue at your end?
>

There are no big spikes, the overall performance seems to be about 40% low.


>
>
>
>
> On Fri, Dec 6, 2013 at 9:10 PM, srmore <comomore@gmail.com> wrote:
>
>>
>>
>>
>> On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak <vicky.kak@gmail.com> wrote:
>>
>>> Hard to say much without knowing about the cassandra configurations.
>>>
>>
>> The cassandra configuration is
>> -Xms8G
>> -Xmx8G
>> -Xmn800m
>> -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC
>> -XX:+CMSParallelRemarkEnabled
>> -XX:SurvivorRatio=4
>> -XX:MaxTenuringThreshold=2
>> -XX:CMSInitiatingOccupancyFraction=75
>> -XX:+UseCMSInitiatingOccupancyOnly
>>
>>
>>
>>> Yes compactions/GC's could skipe the CPU, I had similar behavior with my
>>> setup.
>>>
>>
>> Were you able to get around it ?
>>
>>
>>>
>>> -VK
>>>
>>>
>>> On Fri, Dec 6, 2013 at 7:40 PM, srmore <comomore@gmail.com> wrote:
>>>
>>>> We have a 3 node cluster running cassandra 1.2.12, they are pretty big
>>>> machines 64G ram with 16 cores, cassandra heap is 8G.
>>>>
>>>> The interesting observation is that, when I send traffic to one node
>>>> its performance is 2x more than when I send traffic to all the nodes. We
>>>> ran 1.0.11 on the same box and we observed a slight dip but not half as
>>>> seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM.
>>>> Changing CL to ONE make a slight improvement but not much.
>>>>
>>>> The read_Repair_chance is 0.1. We see some compactions running.
>>>>
>>>> following is my iostat -x output, sda is the ssd (for commit log) and
>>>> sdb is the spinner.
>>>>
>>>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>>>           66.46    0.00    8.95    0.01    0.00   24.58
>>>>
>>>> Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
>>>> avgqu-sz   await  svctm  %util
>>>> sda               0.00    27.60  0.00  4.40     0.00   256.00
>>>> 58.18     0.01    2.55   1.32   0.58
>>>> sda1              0.00     0.00  0.00  0.00     0.00     0.00
>>>> 0.00     0.00    0.00   0.00   0.00
>>>> sda2              0.00    27.60  0.00  4.40     0.00   256.00
>>>> 58.18     0.01    2.55   1.32   0.58
>>>> sdb               0.00     0.00  0.00  0.00     0.00     0.00
>>>> 0.00     0.00    0.00   0.00   0.00
>>>> sdb1              0.00     0.00  0.00  0.00     0.00     0.00
>>>> 0.00     0.00    0.00   0.00   0.00
>>>> dm-0              0.00     0.00  0.00  0.00     0.00     0.00
>>>> 0.00     0.00    0.00   0.00   0.00
>>>> dm-1              0.00     0.00  0.00  0.60     0.00     4.80
>>>> 8.00     0.00    5.33   2.67   0.16
>>>> dm-2              0.00     0.00  0.00  0.00     0.00     0.00
>>>> 0.00     0.00    0.00   0.00   0.00
>>>> dm-3              0.00     0.00  0.00 24.80     0.00   198.40
>>>> 8.00     0.24    9.80   0.13   0.32
>>>> dm-4              0.00     0.00  0.00  6.60     0.00    52.80
>>>> 8.00     0.01    1.36   0.55   0.36
>>>> dm-5              0.00     0.00  0.00  0.00     0.00     0.00
>>>> 0.00     0.00    0.00   0.00   0.00
>>>> dm-6              0.00     0.00  0.00 24.80     0.00   198.40
>>>> 8.00     0.29   11.60   0.13   0.32
>>>>
>>>>
>>>>
>>>> I can see I am cpu bound here but couldn't figure out exactly what is
>>>> causing it, is this caused by GC or Compaction ? I am thinking it is
>>>> compaction, I see a lot of context switches and interrupts in my vmstat
>>>> output.
>>>>
>>>> I don't see GC activity in the logs but see some compaction activity.
>>>> Has anyone seen this ? or know what can be done to free up the CPU.
>>>>
>>>> Thanks,
>>>> Sandeep
>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message