cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "R. Verlangen" <ro...@us2.nl>
Subject Re: Writes slower then reads
Date Thu, 05 Jan 2012 19:53:20 GMT
I turned off 1 large cronjob which caused the CPU not to get used for ~ 60%
once every 10 minutes. Both write and read are fast now. Just think I was
overloading the node.

Weird though that shutting down the node did not improve the speed.

Thank you all for your time!

Robin

2012/1/5 aaron morton <aaron@thelastpickle.com>

> What happens when you turn off the cron jobs ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 6/01/2012, at 6:57 AM, Philippe wrote:
>
> Unless you are doing huge batches no... don't have any other idea for
> now...
>
> 2012/1/5 R. Verlangen <robin@us2.nl>
>
>> The write and read load is very minimal the moment. Roughly 10 writes +
>> 10 reads / second. So 20 operations per second. Don't think that overloads
>> my cluster, does it?
>>
>>
>> 2012/1/5 Philippe <watcherfr@gmail.com>
>>
>>> You may be overloading the cluster though...
>>>
>>> My hypothesis is that your traffic is being spread across your node and
>>> that one slow node is slowing down the fraction of traffic that goes to
>>> that node (when it's acting as coordinator).
>>> So what I would do is reduce the read load a lot to make sure I don't
>>> overload the cluster and measure if I see a 1/RF improvement in response
>>> time which would validate my hypothesis.
>>>
>>>
>>> 2012/1/5 R. Verlangen <robin@us2.nl>
>>>
>>> It does not appear to affect the response time, certainly not in a
>>>> positive way.
>>>>
>>>>
>>>> 2012/1/5 Philippe <watcherfr@gmail.com>
>>>>
>>>>> What if you shutdown the cassandra service on the slow node, does that
>>>>> improve your read performance ?
>>>>> If it does then that sole node is responsible for the slow down
>>>>> because it can't act as a coordinator fast enough.
>>>>>
>>>>> 2012/1/5 R. Verlangen <robin@us2.nl>
>>>>>
>>>>> I'm also reading with CL = ONE
>>>>>>
>>>>>>
>>>>>> 2012/1/5 Philippe <watcherfr@gmail.com>
>>>>>>
>>>>>>> Depending on the CL you're reading at it will yes : if the CL
>>>>>>> requires that the "slow" node create a digest of the data and
send it to
>>>>>>> the coordinator then it might explain the poor performance on
reads. What
>>>>>>> is your read CL ?
>>>>>>>
>>>>>>> 2012/1/5 R. Verlangen <robin@us2.nl>
>>>>>>>
>>>>>>> As I posted this I noticed that the other node's CPU is running
high
>>>>>>>> on some other cronjobs (every couple of minutes to 60% usage).
Is the lack
>>>>>>>> of more CPU cycles a problem in this case?
>>>>>>>>
>>>>>>>> Robin
>>>>>>>>
>>>>>>>> 2012/1/5 R. Verlangen <robin@us2.nl>
>>>>>>>>
>>>>>>>> CPU is idle (< 10% usage). Disk reads occasionally blocks
over
>>>>>>>>> 32/64K. Writes around 0-5MB per second. Network traffic
0.1 / 0.1 MB/s (in
>>>>>>>>> / out). Paging 0. System int ~ 1300, csw ~ 2500.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2012/1/5 Philippe <watcherfr@gmail.com>
>>>>>>>>>
>>>>>>>>>> What can you see in vmstat/dstat ?
>>>>>>>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <robin@us2.nl>
a écrit :
>>>>>>>>>>
>>>>>>>>>> Hi there,
>>>>>>>>>>>
>>>>>>>>>>> I'm running a cassandra 0.8.6 cluster with 2
nodes (in 2 DC's),
>>>>>>>>>>> RF = 2. Actual data on the nodes is only 1GB.
Disk latency < 1ms. Disk
>>>>>>>>>>> throughput ~ 0.4MB/s. OS load always below 1
(on a 8 core machine with 16GB
>>>>>>>>>>> ram).
>>>>>>>>>>>
>>>>>>>>>>> When I'm running my writes against the cluster
with cl = ONE all
>>>>>>>>>>> reads appear to be faster then the writes.
>>>>>>>>>>>
>>>>>>>>>>> Average write speed = 1600us/operation
>>>>>>>>>>> Average read speed = 200us/operation
>>>>>>>>>>>
>>>>>>>>>>> I'm really wondering why this is the case. Anyone
got a clue?
>>>>>>>>>>>
>>>>>>>>>>> With kind regards,
>>>>>>>>>>> Robin
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>

Mime
View raw message