incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Franc Carter <franc.car...@sirca.org.au>
Subject Re: reads/s suddenly dropped
Date Wed, 22 Feb 2012 17:53:06 GMT
On Mon, Feb 20, 2012 at 9:42 PM, Franc Carter <franc.carter@sirca.org.au>wrote:

> On Mon, Feb 20, 2012 at 12:00 PM, aaron morton <aaron@thelastpickle.com>wrote:
>
>> Aside from iostats..
>>
>> nodetool cfstats will give you read and write latency for each CF. This
>> is the latency for the operation on each node. Check that to see if latency
>> is increasing.
>>
>> Take a look at nodetool compactionstats to see if compactions are running
>> at the same time. The IO is throttled but if you are on aws it may not be
>> throttled enough.
>>
>>
> compaction had finished
>
>
>> The sweet spot for non netflix deployments seems to be a m1.xlarge with
>> 16GB. THe JVM can have 8 and the rest can be used for memmapping files.
>> Here is a good post about choosing EC2 sizes…
>> http://perfcap.blogspot.co.nz/2011/03/understanding-and-using-amazon-ebs.html
>>
>
> Thanks - good article. I'll go up to m1.xlarge and explore that behaviour
>

the m1.xlarge is giving much better and more consistent results

thanks


>
> cheers
>
>
>
>>
>> Cheers
>>
>>   -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 20/02/2012, at 9:31 AM, Franc Carter wrote:
>>
>> On Mon, Feb 20, 2012 at 4:10 AM, Philippe <watcherfr@gmail.com> wrote:
>>
>>> Perhaps your dataset can no longer be held in memory. Check iostats
>>>
>>
>> I have been flushing the keycache and dropping the linux disk caches
>> before each to avoid testing memory reads.
>>
>> One possibility that I thought of is that the success keys are now 'far
>> enough away' that they are not being included in the previous read and
>> hence the seek penalty has to be paid a lot more often  - viable ?
>>
>> cheers
>>
>>>
>>> Le 19 févr. 2012 11:24, "Franc Carter" <franc.carter@sirca.org.au> a
>>> écrit :
>>>
>>>
>>>> I've been testing Cassandra - primarily looking at reads/second for our
>>>> fairly data model - one unique key with a row of columns that we always
>>>> request. I've now setup the cluster with with m1.large (2 cpus 8GB)
>>>>
>>>> I had loaded a months worth of data in and was doing random requests as
>>>> a torture test - and getting very nice results. I then loaded another days
>>>> worth of day and repeated the tests while the load was running - still good.
>>>>
>>>> I then started loading more days and at some point the performance
>>>> dropped by close to an order of magnitude ;-(
>>>>
>>>> Any ideas on what to look for ?
>>>>
>>>> thanks
>>>>
>>>> --
>>>> *Franc Carter* | Systems architect | Sirca Ltd
>>>>  <marc.zianideferranti@sirca.org.au>
>>>> franc.carter@sirca.org.au | www.sirca.org.au
>>>> Tel: +61 2 9236 9118
>>>>  Level 9, 80 Clarence St, Sydney NSW 2000
>>>> PO Box H58, Australia Square, Sydney NSW 1215
>>>>
>>>>
>>
>>
>> --
>> *Franc Carter* | Systems architect | Sirca Ltd
>>  <marc.zianideferranti@sirca.org.au>
>> franc.carter@sirca.org.au | www.sirca.org.au
>> Tel: +61 2 9236 9118
>>  Level 9, 80 Clarence St, Sydney NSW 2000
>> PO Box H58, Australia Square, Sydney NSW 1215
>>
>>
>>
>
>
> --
>
> *Franc Carter* | Systems architect | Sirca Ltd
>  <marc.zianideferranti@sirca.org.au>
>
> franc.carter@sirca.org.au | www.sirca.org.au
>
> Tel: +61 2 9236 9118
>
> Level 9, 80 Clarence St, Sydney NSW 2000
>
> PO Box H58, Australia Square, Sydney NSW 1215
>
>


-- 

*Franc Carter* | Systems architect | Sirca Ltd
 <marc.zianideferranti@sirca.org.au>

franc.carter@sirca.org.au | www.sirca.org.au

Tel: +61 2 9236 9118

Level 9, 80 Clarence St, Sydney NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215

Mime
View raw message