Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 74.125.82.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=Kkq1cLsfrb800vTLFI+1HkJpR7VgwYyU1ykwIf8MkF6LHbzsjM8OPVlOaC3u3zppx0
         K0Ii1ScOV+llPjaAYdnhMlvRRj2gnZoX3O0JC7KgkDWmZhOiKREE5oj5mxiHofKvfdCJ
         ZkY3hC9/+B709rzWslW/2K442/CqbpzIIxi0A=
MIME-Version: 1.0
In-Reply-To: <g2ie4b4609b1005061056k778f5f6cz29847cde9a718970@mail.gmail.com>
References: <h2xe4b4609b1005032320x48699668qacef6db0e91c7011@mail.gmail.com>
	<i2ze06563881005041227sbd857759saa699b9ff2f97084@mail.gmail.com>
	<8A606DEA-CB57-4D0B-90C0-FE79B2DE22E9@discovereads.com>
 <m2xe4b4609b1005050819j1d4e190en2cb1fabcc466c90e@mail.gmail.com>
	<o2ye06563881005050908wd93315f7x41cc6aca0889d583@mail.gmail.com>
	<w2pe4b4609b1005051459o5c35480crf9c58bf8e23c5487@mail.gmail.com>
	<t2ue06563881005051825ga1e0bb48re95bb1694e0e4b28@mail.gmail.com>
	<r2me4b4609b1005052130yea6aeef0y6245ad83ef5eb32e@mail.gmail.com>
	<r2ue06563881005060722s2abbe24aqba7120a839b61c42@mail.gmail.com>
	<g2ie4b4609b1005061056k778f5f6cz29847cde9a718970@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Thu, 6 May 2010 16:39:50 -0500
Message-ID: <s2ue06563881005061439y19f0b0f8g9624237c1e41806b@mail.gmail.com>
Subject: Re: performance tuning - where does the slowness come from?
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Yes, that makes sense.  If you never have a warm cache then it's
probably disk seek time creating that latency, in which case there
isn't a whole lot you can do about it short of adding more capacity
(so at least it's cached at the OS level).

iostat -x could substantiate this guess.

On Thu, May 6, 2010 at 12:56 PM, Ran Tavory <rantav@gmail.com> wrote:
> Jonathan, I think it's the case of large values in the columns. The
> problematic CF is a key-value store, so it has only one column per row,
> however the value of that column can be large. It's a java serialized obj=
ect
> (uncompressed) which, may be 100s of bytes, maybe even a few megs. This C=
F
> also suffers from zero cache hits since each time a read is for a unique
> key.
> I ran stress.py and I see much better=A0results=A0(reads are < 1ms) so I =
assume
> my cluster is=A0healthy, so I need to fix the app. Would 1meg bytes objec=
t
> explain a 30ms (sometimes even more) read latency? The boxes aren't fancy=
,
> not sure exactly what hardware we have there but it's "commodity"...
> Thanks!
>
> On Thu, May 6, 2010 at 5:22 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> columns, not CFs.
>>
>> put another way, how wide are the rows in the slow CF?
>>
>> On Wed, May 5, 2010 at 11:30 PM, Ran Tavory <rantav@gmail.com> wrote:
>> > I have a few CFs but the one I'm seeing slowness in, which is the one
>> > with
>> > plenty of cache misses has only one column per key.
>> > Latency varies b/w 10m and 60ms but I'd say average is 30ms.
>> >
>> > On Thu, May 6, 2010 at 4:25 AM, Jonathan Ellis <jbellis@gmail.com>
>> > wrote:
>> >>
>> >> How many columns are in the rows you are reading from?
>> >>
>> >> 30ms is quite high, so I suspect you have relatively large rows, in
>> >> which case decreasing the column index threshold may help.
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com