cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: reads/s suddenly dropped
Date Mon, 20 Feb 2012 01:00:15 GMT
Aside from iostats..

nodetool cfstats will give you read and write latency for each CF. This is the latency for
the operation on each node. Check that to see if latency is increasing. 

Take a look at nodetool compactionstats to see if compactions are running at the same time.
The IO is throttled but if you are on aws it may not be throttled enough. 

The sweet spot for non netflix deployments seems to be a m1.xlarge with 16GB. THe JVM can
have 8 and the rest can be used for memmapping files. Here is a good post about choosing EC2
sizes… http://perfcap.blogspot.co.nz/2011/03/understanding-and-using-amazon-ebs.html

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 20/02/2012, at 9:31 AM, Franc Carter wrote:

> On Mon, Feb 20, 2012 at 4:10 AM, Philippe <watcherfr@gmail.com> wrote:
> Perhaps your dataset can no longer be held in memory. Check iostats
> 
> 
> I have been flushing the keycache and dropping the linux disk caches before each to avoid
testing memory reads.
> 
> One possibility that I thought of is that the success keys are now 'far enough away'
that they are not being included in the previous read and hence the seek penalty has to be
paid a lot more often  - viable ?
>  
> cheers
> 
> Le 19 févr. 2012 11:24, "Franc Carter" <franc.carter@sirca.org.au> a écrit :
> 
> 
> I've been testing Cassandra - primarily looking at reads/second for our fairly data model
- one unique key with a row of columns that we always request. I've now setup the cluster
with with m1.large (2 cpus 8GB)
> 
> I had loaded a months worth of data in and was doing random requests as a torture test
- and getting very nice results. I then loaded another days worth of day and repeated the
tests while the load was running - still good.
> 
> I then started loading more days and at some point the performance dropped by close to
an order of magnitude ;-(
> 
> Any ideas on what to look for ?
> 
> thanks
> 
> -- 
> Franc Carter | Systems architect | Sirca Ltd
> franc.carter@sirca.org.au | www.sirca.org.au
> Tel: +61 2 9236 9118 
> Level 9, 80 Clarence St, Sydney NSW 2000
> PO Box H58, Australia Square, Sydney NSW 1215
> 
> 
> 
> 
> -- 
> Franc Carter | Systems architect | Sirca Ltd
> franc.carter@sirca.org.au | www.sirca.org.au
> Tel: +61 2 9236 9118 
> Level 9, 80 Clarence St, Sydney NSW 2000
> PO Box H58, Australia Square, Sydney NSW 1215
> 


Mime
View raw message