incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Franc Carter <franc.car...@sirca.org.au>
Subject Re: reads/s suddenly dropped
Date Mon, 20 Feb 2012 10:42:59 GMT
On Mon, Feb 20, 2012 at 12:00 PM, aaron morton <aaron@thelastpickle.com>wrote:

> Aside from iostats..
>
> nodetool cfstats will give you read and write latency for each CF. This is
> the latency for the operation on each node. Check that to see if latency is
> increasing.
>
> Take a look at nodetool compactionstats to see if compactions are running
> at the same time. The IO is throttled but if you are on aws it may not be
> throttled enough.
>
>
compaction had finished


> The sweet spot for non netflix deployments seems to be a m1.xlarge with
> 16GB. THe JVM can have 8 and the rest can be used for memmapping files.
> Here is a good post about choosing EC2 sizes…
> http://perfcap.blogspot.co.nz/2011/03/understanding-and-using-amazon-ebs.html
>

Thanks - good article. I'll go up to m1.xlarge and explore that behaviour

cheers



>
> Cheers
>
>   -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/02/2012, at 9:31 AM, Franc Carter wrote:
>
> On Mon, Feb 20, 2012 at 4:10 AM, Philippe <watcherfr@gmail.com> wrote:
>
>> Perhaps your dataset can no longer be held in memory. Check iostats
>>
>
> I have been flushing the keycache and dropping the linux disk caches
> before each to avoid testing memory reads.
>
> One possibility that I thought of is that the success keys are now 'far
> enough away' that they are not being included in the previous read and
> hence the seek penalty has to be paid a lot more often  - viable ?
>
> cheers
>
>>
>> Le 19 févr. 2012 11:24, "Franc Carter" <franc.carter@sirca.org.au> a
>> écrit :
>>
>>
>>> I've been testing Cassandra - primarily looking at reads/second for our
>>> fairly data model - one unique key with a row of columns that we always
>>> request. I've now setup the cluster with with m1.large (2 cpus 8GB)
>>>
>>> I had loaded a months worth of data in and was doing random requests as
>>> a torture test - and getting very nice results. I then loaded another days
>>> worth of day and repeated the tests while the load was running - still good.
>>>
>>> I then started loading more days and at some point the performance
>>> dropped by close to an order of magnitude ;-(
>>>
>>> Any ideas on what to look for ?
>>>
>>> thanks
>>>
>>> --
>>> *Franc Carter* | Systems architect | Sirca Ltd
>>>  <marc.zianideferranti@sirca.org.au>
>>> franc.carter@sirca.org.au | www.sirca.org.au
>>> Tel: +61 2 9236 9118
>>>  Level 9, 80 Clarence St, Sydney NSW 2000
>>> PO Box H58, Australia Square, Sydney NSW 1215
>>>
>>>
>
>
> --
> *Franc Carter* | Systems architect | Sirca Ltd
>  <marc.zianideferranti@sirca.org.au>
> franc.carter@sirca.org.au | www.sirca.org.au
> Tel: +61 2 9236 9118
>  Level 9, 80 Clarence St, Sydney NSW 2000
> PO Box H58, Australia Square, Sydney NSW 1215
>
>
>


-- 

*Franc Carter* | Systems architect | Sirca Ltd
 <marc.zianideferranti@sirca.org.au>

franc.carter@sirca.org.au | www.sirca.org.au

Tel: +61 2 9236 9118

Level 9, 80 Clarence St, Sydney NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215

Mime
View raw message