Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7A9739106 for ; Mon, 13 Feb 2012 09:03:06 +0000 (UTC) Received: (qmail 5940 invoked by uid 500); 13 Feb 2012 09:03:03 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 5757 invoked by uid 500); 13 Feb 2012 09:02:48 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 5745 invoked by uid 99); 13 Feb 2012 09:02:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Feb 2012 09:02:45 +0000 X-ASF-Spam-Status: No, hits=0.6 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_NEUTRAL,TO_NO_BRKTS_PCNT X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [74.125.149.151] (HELO na3sys009aog124.obsmtp.com) (74.125.149.151) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 13 Feb 2012 09:02:38 +0000 Received: from mail-lpp01m010-f41.google.com ([209.85.215.41]) (using TLSv1) by na3sys009aob124.postini.com ([74.125.148.12]) with SMTP ID DSNKTzjRl90uSXWF40WboMRLCJjNU6u80cwZ@postini.com; Mon, 13 Feb 2012 01:02:17 PST Received: by lamf4 with SMTP id f4so4037166lam.28 for ; Mon, 13 Feb 2012 01:02:14 -0800 (PST) Received: by 10.112.99.106 with SMTP id ep10mr5367287lbb.68.1329123404178; Mon, 13 Feb 2012 00:56:44 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.26.199 with HTTP; Mon, 13 Feb 2012 00:56:24 -0800 (PST) In-Reply-To: References: <201202131403484201571@jike.com> From: Franc Carter Date: Mon, 13 Feb 2012 19:56:24 +1100 Message-ID: Subject: Re: keycache persisted to disk ? To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d040169c75e646604b8d4a828 X-Gm-Message-State: ALoCoQkMu6vLh/K/EeEQe0KcGLzZj6mkUvL/YmnXFpKdQ4XmxMNtIEYCXOYloc7gqdVi+LHHqH0K X-Virus-Checked: Checked by ClamAV on apache.org --f46d040169c75e646604b8d4a828 Content-Type: text/plain; charset=ISO-8859-1 On Mon, Feb 13, 2012 at 7:48 PM, Peter Schuller wrote: > > Yep - I've been looking at these - I don't see anything in iostat/dstat > etc > > that point strongly to a problem. There is quite a bit of I/O load, but > it > > looks roughly uniform on slow and fast instances of the queries. The last > > compaction ran 4 days ago - which was before I started seeing variable > > performance > > [snip] > > > I now why it is slow - it's clearly I/O bound. I am trying to hunt down > why > > it is sometimes much faster even though I have (tried) to replicate the > > same conditions > > What does clearly I/O bound mean, and what is "quite a bit" of I/O > load? the servers spending >50% of the time in io-wait > In general, if you have queries that come in at some rate that > is determined by outside sources (rather than by the time the last > query took to execute), That's an interesting approach - is that likely to give close to optimal performance ? > you will typically either get more queries > than your cluster can take, or fewer. If fewer, there is a > non-trivially sized grey area where overall I/O throughput needed is > lower than that available, but the closer you are to capacity the more > often requests have to wait for other I/O to complete, for purely > statistical reasons. > > If you're running close to maximum capacity, it would be expected that > the variation in query latency is high. > That may well explain it - I'll have to think about what that means for our use case as load will be extremely bursty > > That said, if you're seeing consistently bad latencies for a while > where you sometimes see consistently good latencies, that sounds > different but would hopefully be observable somehow. > > -- > / Peter Schuller (@scode, http://worldmodscode.wordpress.com) > -- *Franc Carter* | Systems architect | Sirca Ltd franc.carter@sirca.org.au | www.sirca.org.au Tel: +61 2 9236 9118 Level 9, 80 Clarence St, Sydney NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215 --f46d040169c75e646604b8d4a828 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Mon, Feb 13, 2012 at 7:48 PM, Peter Schuller <peter.schuller@infidyne.com&g= t; wrote:
> Yep - I've been looking at these - I don't s= ee anything in iostat/dstat etc
> that point strongly to a problem. There is quite a bit of I/O load, bu= t it
> looks roughly uniform on slow and fast instances of the queries. The l= ast
> compaction ran 4 days ago - which was before I started seeing variable=
> performance

[snip]

> I now why it is slow - it's clearly I/O bound. I am trying to hunt= down why
> it is sometimes much faster even though I have (tried) to replicate = =A0the
> same conditions

What does clearly I/O bound mean, and what is "quite a bit"= of I/O
load?

the servers spending >50% of the t= ime in io-wait
=A0
In gene= ral, if you have queries that come in at some rate that
is determined by outside sources (rather than by the time the last
query took to execute),

That's an inter= esting approach - is that likely to give close to optimal performance ?
=A0
you will typically either get more queries
than your cluster can take, or fewer. If fewer, there is a
non-trivially sized grey area where overall I/O throughput needed is
lower than that available, but the closer you are to capacity the more
often requests have to wait for other I/O to complete, for purely
statistical reasons.

If you're running close to maximum capacity, it would be expected that<= br> the variation in query latency is high.

That may well explain it - I'll have to think about what that means fo= r our use case as load will be extremely bursty=A0
=A0

That said, if you're seeing consistently bad latencies for a while
where you sometimes see consistently good latencies, that sounds
different but would hopefully be observable somehow.

--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)



--

Franc Carter<= /b> |<= /span> Systems architect | Sirca Ltd

franc.carter@sirca.org.au=A0|=A0www.sirca.org.au

Tel:= =A0+61 2 9236 9118

Level 9, 80 Clarence St, Sydney=A0NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215<= /span>


--f46d040169c75e646604b8d4a828--