cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thorsten von Eicken <...@rightscale.com>
Subject Re: Persistently increasing read latency
Date Sat, 05 Dec 2009 04:40:51 GMT
Jonathan Ellis wrote:
> On Fri, Dec 4, 2009 at 1:31 PM, Freeman, Tim <tim.freeman@hp.com> wrote:
>   
>>> Fundamentally there's only so much I/O you can do at a time.  If you
>>> don't have enough, you need to upgrade to servers with better i/o
>>> (i.e. not EC2: http://pl.atyp.us/wordpress/?p=2240&cpage=1) and/or
>>> more ram to cache the reads against.
>>>       
>> I agree that any given configuration will support a limited amount of I/O.
>>
>> For the first few hours of my load test, I have enough I/O.  The problem is that
Cassandra is spending too much I/O on reads and writes and too little on compactions to function
well in the long term.
>>     
>
> If you don't have enough room for both, it doesn't matter how you prioritize.
>   
Mhhh, maybe... You're technically correct. The question here is whether 
cassandra degrades gracefully or not. If I understand correctly, there 
are two ways to look at it:

1) it's accepting a higher request load than it can actually process and 
builds up an increasing backlog that eventually brings performance down 
far below the level of performance that it could sustain, thus it fails 
to do the type of early admission control or back-pressure that keeps 
the request load close to the sustainable maximum performance.

2) the compaction backlog size is a primary variable that has to be 
exposed and monitored in any cassandra installation because it's a 
direct indicator for an overload situation, just like hitting 100% cpu 
or similar would be.

I can buy that (2) is ok, but (1) is certainly nicer.
Thorsten

Mime
View raw message