Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of weijunli@gmail.com designates
 209.85.222.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=pTRh+hXX+11SdB9wcl5LF7fIvRFdlhhGQqCRuN3NvdMBo3jmT+EMmmPWYc2mUfcM8G
         O3J9Pc+Hs9Cb/rbbRQYZwo82LM/T5Zeym0qGzYO6uRQbG1slXi1lPcPjtPDWy5DVUmUu
         0bU1QYeDsvjKZnosakXXMM72BxsvRRaszvZ5A=
MIME-Version: 1.0
In-Reply-To: <cdc5ad201002161001t1c8e5409kd2d308819e009e39@mail.gmail.com>
References: <468b21171001200244n2521e77esa84964946f0eb20b@mail.gmail.com>
	 <e4b4609b1002132210h4b79bc52mb50bc7eb53e2ad88@mail.gmail.com>
	 <012601caade5$21a79fb0$64f6df10$@com>
	 <e06563881002141822n230503f6m17ec3afca0e4a569@mail.gmail.com>
	 <005101caaedd$2bb3ce90$831b6bb0$@com>
	 <1NhIrQ-0007Jt-In@mail.eleven.de>
	 <cdc5ad201002160606o5a6ecf3fl1ec5d7eb0d2e2e0@mail.gmail.com>
	 <ad2266761002160950q3f4c48afv6153e06a5f2e3b2a@mail.gmail.com>
	 <ad2266761002160956y6657f7as65c2315ae8a81bf7@mail.gmail.com>
	 <cdc5ad201002161001t1c8e5409kd2d308819e009e39@mail.gmail.com>
Date: Tue, 16 Feb 2010 10:16:35 -0800
Message-ID: <ad2266761002161016v2a1cf4b8r69d3057c4a77d70@mail.gmail.com>
Subject: Re: Cassandra benchmark shows OK throughput but high read latency (>
	100ms)?
From: Weijun Li <weijunli@gmail.com>
To: cassandra-user@incubator.apache.org
Content-Type: multipart/alternative; boundary=0016e64de750f53325047fbbbbb8

--0016e64de750f53325047fbbbbb8
Content-Type: text/plain; charset=ISO-8859-1

Thanks for for DataFileDirectory trick and I'll give a try.

Just noticed the impact of number of data files: node A has 13 data files
with read latency of 20ms and node B has 27 files with read latency of 60ms.
After I ran "nodeprobe compact" on node B its read latency went up to 150ms.
The read latency of node A became as low as 10ms. Is this normal behavior?
I'm using random partitioner and the hardware/JVM settings are exactly the
same for these two nodes.

Another problem is that Java heap usage is always 900mb out of 6GB? Is there
any way to utilize all of the heap space to decrease the read latency?

-Weijun

On Tue, Feb 16, 2010 at 10:01 AM, Brandon Williams <driftx@gmail.com> wrote:

> On Tue, Feb 16, 2010 at 11:56 AM, Weijun Li <weijunli@gmail.com> wrote:
>
>> One more thoughts about Martin's suggestion: is it possible to put the
>> data files into multiple directories that are located in different physical
>> disks? This should help to improve the i/o bottleneck issue.
>>
>>
> Yes, you can already do this, just add more <DataFileDirectory> directives
> pointed at multiple drives.
>
>
>> Has anybody tested the row-caching feature in trunk (shoot for 0.6?)?
>
>
> Row cache and key cache both help tremendously if your read pattern has a
> decent repeat rate.  Completely random io can only be so fast, however.
>
> -Brandon
>

--0016e64de750f53325047fbbbbb8
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks for for DataFileDirectory trick and I&#39;ll give a try.<br><br>Just=
 noticed the impact of number of data files: node A has 13 data files with =
read latency of 20ms and node B has 27 files with read latency of 60ms. Aft=
er I ran &quot;nodeprobe compact&quot; on node B its read latency went up t=
o 150ms. The read latency of node A became as low as 10ms. Is this normal b=
ehavior? I&#39;m using random partitioner and the hardware/JVM settings are=
 exactly the same for these two nodes.<br>
<br>Another problem is that Java heap usage is always 900mb out of 6GB? Is =
there any way to utilize all of the heap space to decrease the read latency=
?<br><br>-Weijun<br><br><div class=3D"gmail_quote">On Tue, Feb 16, 2010 at =
10:01 AM, Brandon Williams <span dir=3D"ltr">&lt;<a href=3D"mailto:driftx@g=
mail.com">driftx@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class=3D"gma=
il_quote"><div class=3D"im">On Tue, Feb 16, 2010 at 11:56 AM, Weijun Li <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:weijunli@gmail.com" target=3D"_blank">=
weijunli@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
One more thoughts about Martin&#39;s suggestion: is it possible to put the =
data files into multiple directories that are located in different physical=
 disks? This should help to improve the i/o bottleneck issue.<br><br></bloc=
kquote>

<div><br></div></div><div>Yes, you can already do this, just add more=A0&lt=
;DataFileDirectory&gt; directives pointed at multiple drives.</div><div cla=
ss=3D"im"><div>=A0</div><blockquote class=3D"gmail_quote" style=3D"border-l=
eft: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:=
 1ex;">

Has anybody tested the row-caching feature in trunk (shoot for 0.6?)?</bloc=
kquote><div><br></div></div><div>Row cache and key cache both help tremendo=
usly if your read pattern has a decent repeat rate. =A0Completely random io=
 can only be so fast, however.</div>

<div><br></div><font color=3D"#888888"><div>-Brandon=A0</div></font></div>
</blockquote></div><br>

--0016e64de750f53325047fbbbbb8--