cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Writes slower then reads
Date Thu, 05 Jan 2012 19:20:06 GMT
What happens when you turn off the cron jobs ? 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 6/01/2012, at 6:57 AM, Philippe wrote:

> Unless you are doing huge batches no... don't have any other idea for now...
> 
> 2012/1/5 R. Verlangen <robin@us2.nl>
> The write and read load is very minimal the moment. Roughly 10 writes + 10 reads / second.
So 20 operations per second. Don't think that overloads my cluster, does it?
> 
> 
> 2012/1/5 Philippe <watcherfr@gmail.com>
> You may be overloading the cluster though...
> 
> My hypothesis is that your traffic is being spread across your node and that one slow
node is slowing down the fraction of traffic that goes to that node (when it's acting as coordinator).
> So what I would do is reduce the read load a lot to make sure I don't overload the cluster
and measure if I see a 1/RF improvement in response time which would validate my hypothesis.
> 
> 
> 2012/1/5 R. Verlangen <robin@us2.nl>
> 
> It does not appear to affect the response time, certainly not in a positive way.
> 
> 
> 2012/1/5 Philippe <watcherfr@gmail.com>
> What if you shutdown the cassandra service on the slow node, does that improve your read
performance ?
> If it does then that sole node is responsible for the slow down because it can't act
as a coordinator fast enough.
> 
> 2012/1/5 R. Verlangen <robin@us2.nl>
> 
> I'm also reading with CL = ONE
> 
> 
> 2012/1/5 Philippe <watcherfr@gmail.com>
> Depending on the CL you're reading at it will yes : if the CL requires that the "slow"
node create a digest of the data and send it to the coordinator then it might explain the
poor performance on reads. What is your read CL ?
> 
> 2012/1/5 R. Verlangen <robin@us2.nl>
> 
> As I posted this I noticed that the other node's CPU is running high on some other cronjobs
(every couple of minutes to 60% usage). Is the lack of more CPU cycles a problem in this case?
> 
> Robin
> 
> 2012/1/5 R. Verlangen <robin@us2.nl>
> 
> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. Writes around
0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). Paging 0. System int ~ 1300,
csw ~ 2500.
> 
> 
> 2012/1/5 Philippe <watcherfr@gmail.com>
> What can you see in vmstat/dstat ?
> 
> Le 5 janv. 2012 11:58, "R. Verlangen" <robin@us2.nl> a écrit :
> 
> Hi there,
> 
> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. Actual data on
the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ 0.4MB/s. OS load always below
1 (on a 8 core machine with 16GB ram). 
> 
> When I'm running my writes against the cluster with cl = ONE all reads appear to be faster
then the writes. 
> 
> Average write speed = 1600us/operation
> Average read speed = 200us/operation
> 
> I'm really wondering why this is the case. Anyone got a clue?
> 
> With kind regards,
> Robin 
> 
> 
> 
> 
> 
> 
> 
> 
> 


Mime
View raw message