incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Franc Carter <franc.car...@sirca.org.au>
Subject Re: keycache persisted to disk ?
Date Mon, 13 Feb 2012 08:56:24 GMT
On Mon, Feb 13, 2012 at 7:48 PM, Peter Schuller <peter.schuller@infidyne.com
> wrote:

> > Yep - I've been looking at these - I don't see anything in iostat/dstat
> etc
> > that point strongly to a problem. There is quite a bit of I/O load, but
> it
> > looks roughly uniform on slow and fast instances of the queries. The last
> > compaction ran 4 days ago - which was before I started seeing variable
> > performance
>
> [snip]
>
> > I now why it is slow - it's clearly I/O bound. I am trying to hunt down
> why
> > it is sometimes much faster even though I have (tried) to replicate  the
> > same conditions
>
> What does clearly I/O bound mean, and what is "quite a bit" of I/O
> load?


the servers spending >50% of the time in io-wait


> In general, if you have queries that come in at some rate that
> is determined by outside sources (rather than by the time the last
> query took to execute),


That's an interesting approach - is that likely to give close to optimal
performance ?


> you will typically either get more queries
> than your cluster can take, or fewer. If fewer, there is a
> non-trivially sized grey area where overall I/O throughput needed is
> lower than that available, but the closer you are to capacity the more
> often requests have to wait for other I/O to complete, for purely
> statistical reasons.
>
> If you're running close to maximum capacity, it would be expected that
> the variation in query latency is high.
>

That may well explain it - I'll have to think about what that means for our
use case as load will be extremely bursty


>
> That said, if you're seeing consistently bad latencies for a while
> where you sometimes see consistently good latencies, that sounds
> different but would hopefully be observable somehow.
>
> --
> / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
>



-- 

*Franc Carter* | Systems architect | Sirca Ltd
 <marc.zianideferranti@sirca.org.au>

franc.carter@sirca.org.au | www.sirca.org.au

Tel: +61 2 9236 9118

Level 9, 80 Clarence St, Sydney NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215

Mime
View raw message