cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe <watche...@gmail.com>
Subject Re: Writes slower then reads
Date Thu, 05 Jan 2012 17:57:10 GMT
Unless you are doing huge batches no... don't have any other idea for now...

2012/1/5 R. Verlangen <robin@us2.nl>

> The write and read load is very minimal the moment. Roughly 10 writes + 10
> reads / second. So 20 operations per second. Don't think that overloads my
> cluster, does it?
>
>
> 2012/1/5 Philippe <watcherfr@gmail.com>
>
>> You may be overloading the cluster though...
>>
>> My hypothesis is that your traffic is being spread across your node and
>> that one slow node is slowing down the fraction of traffic that goes to
>> that node (when it's acting as coordinator).
>> So what I would do is reduce the read load a lot to make sure I don't
>> overload the cluster and measure if I see a 1/RF improvement in response
>> time which would validate my hypothesis.
>>
>>
>> 2012/1/5 R. Verlangen <robin@us2.nl>
>>
>> It does not appear to affect the response time, certainly not in a
>>> positive way.
>>>
>>>
>>> 2012/1/5 Philippe <watcherfr@gmail.com>
>>>
>>>> What if you shutdown the cassandra service on the slow node, does that
>>>> improve your read performance ?
>>>> If it does then that sole node is responsible for the slow down because
>>>> it can't act as a coordinator fast enough.
>>>>
>>>> 2012/1/5 R. Verlangen <robin@us2.nl>
>>>>
>>>> I'm also reading with CL = ONE
>>>>>
>>>>>
>>>>> 2012/1/5 Philippe <watcherfr@gmail.com>
>>>>>
>>>>>> Depending on the CL you're reading at it will yes : if the CL
>>>>>> requires that the "slow" node create a digest of the data and send
it to
>>>>>> the coordinator then it might explain the poor performance on reads.
What
>>>>>> is your read CL ?
>>>>>>
>>>>>> 2012/1/5 R. Verlangen <robin@us2.nl>
>>>>>>
>>>>>> As I posted this I noticed that the other node's CPU is running high
>>>>>>> on some other cronjobs (every couple of minutes to 60% usage).
Is the lack
>>>>>>> of more CPU cycles a problem in this case?
>>>>>>>
>>>>>>> Robin
>>>>>>>
>>>>>>> 2012/1/5 R. Verlangen <robin@us2.nl>
>>>>>>>
>>>>>>> CPU is idle (< 10% usage). Disk reads occasionally blocks
over
>>>>>>>> 32/64K. Writes around 0-5MB per second. Network traffic 0.1
/ 0.1 MB/s (in
>>>>>>>> / out). Paging 0. System int ~ 1300, csw ~ 2500.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2012/1/5 Philippe <watcherfr@gmail.com>
>>>>>>>>
>>>>>>>>> What can you see in vmstat/dstat ?
>>>>>>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <robin@us2.nl>
a écrit :
>>>>>>>>>
>>>>>>>>> Hi there,
>>>>>>>>>>
>>>>>>>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes
(in 2 DC's),
>>>>>>>>>> RF = 2. Actual data on the nodes is only 1GB. Disk
latency < 1ms. Disk
>>>>>>>>>> throughput ~ 0.4MB/s. OS load always below 1 (on
a 8 core machine with 16GB
>>>>>>>>>> ram).
>>>>>>>>>>
>>>>>>>>>> When I'm running my writes against the cluster with
cl = ONE all
>>>>>>>>>> reads appear to be faster then the writes.
>>>>>>>>>>
>>>>>>>>>> Average write speed = 1600us/operation
>>>>>>>>>> Average read speed = 200us/operation
>>>>>>>>>>
>>>>>>>>>> I'm really wondering why this is the case. Anyone
got a clue?
>>>>>>>>>>
>>>>>>>>>> With kind regards,
>>>>>>>>>> Robin
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message