Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Chris Goffinet <cg@chrisgoffinet.com>
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: multipart/alternative; boundary=Apple-Mail-1-885161600
Subject: Re: Read Latency Degradation
Date: Sat, 18 Dec 2010 18:16:14 -0800
In-Reply-To: <AANLkTinSGyTAKSxq9Q2EWYUjqdOqEPw8jSfW9iiPEPJQ@mail.gmail.com>
To: user@cassandra.apache.org
References: <AANLkTim-ca44qcHMfSao4=j8QTgL3dP6xSQPdAN0vWgf@mail.gmail.com>
 <5B3E740C-DB61-47AE-A8D6-9D8086A01422@gmx.net>
 <AANLkTikD74JoDngm7mK5_9VCNafViAxt+_qYvy9gEkyT@mail.gmail.com>
 <AANLkTinSMyJ2y0steHJT93_jTMEoU0Yure3_biTm5VdN@mail.gmail.com>
 <AANLkTinpK7OPfd9hOStpT6c5W10veSbAK+8tkFJF-h_B@mail.gmail.com>
 <AANLkTimB3yU+BKeZDU428csdmZgHuUA0nr3KBk5hu5YF@mail.gmail.com>
 <AANLkTik1frr0C5RoNz=ndopr-f=nkMywr2JzXjpizwqy@mail.gmail.com>
 <AANLkTimeveQb=horMk-_RJtMghDDAaQoCrtSiqL=F-Wf@mail.gmail.com>
 <AANLkTinJXYZ7kwd0kFpq2e8eRvmcSjg0y7rEW3tKsOkT@mail.gmail.com>
 <AANLkTinEtXnwTJUQTkm2FUmNE9NdBUkMfEPyQJAibS0u@mail.gmail.com>
 <AANLkTinSGyTAKSxq9Q2EWYUjqdOqEPw8jSfW9iiPEPJQ@mail.gmail.com>
Message-Id: <3DAB334F-86CE-4F85-BFEE-B4D29E56507A@chrisgoffinet.com>


--Apple-Mail-1-885161600
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

You can disable compaction and enable it later. Use nodetool and =
setcompactionthreshold to 0 0

-Chris

On Dec 18, 2010, at 6:05 PM, Wayne wrote:

> Rereading through everything again I am starting to wonder if the page =
cache is being affected by compaction. We have been heavily loading data =
for weeks and compaction is basically running non-stop. The manual =
compaction should be done some time tomorrow, so when totally caught up =
I will try again. What changes can be hoped for in 1470 or 1882 in terms =
of isolating compactions (or writes) affects on read requests?
>=20
> Thanks
>=20
>=20
>=20
> On Sat, Dec 18, 2010 at 2:36 PM, Peter Schuller =
<peter.schuller@infidyne.com> wrote:
> > You are absolutely back to my main concern. Initially we were =
consistently
> > seeing < 10ms read latency and now we see 25ms (30GB sstable file), =
50ms
> > (100GB sstable file) and 65ms (330GB table file) read times for a =
single
> > read with nothing else going on in the cluster. Concurrency is not =
our
> > problem/concern (at this point), our problem is slow reads in total
> > isolation. Frankly the concern is that a 2TB node with a 1TB sstable =
(worst
> > case scenario) will result in > 100ms read latency in total =
isolation.
>=20
> So if you have a single non-concurrent client, along, submitting these
> reads that take 65 ms - are you disk bound (according to the last
> column of iostat -x 1), and how many reads per second (rps column) are
> you seeing relative to client reads? Is the number of disk reads per
> client read consistent with the actual number of sstables at the time?
>=20
> The behavior you're describing really does seem indicative of a
> problem, unless the the bottleneck is legitimately reads from disk
> from multiple sstables resulting from rows being spread over said
> sstables.
>=20
> --
> / Peter Schuller
>=20


--Apple-Mail-1-885161600
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">You can disable compaction and enable it later. Use nodetool and setcompactionthreshold to 0 0<div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div>-Chris</div></span>
</div>
<br><div><div>On Dec 18, 2010, at 6:05 PM, Wayne wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Rereading through everything again I am starting to wonder if the page cache is being affected by compaction. We have been heavily loading data for weeks and compaction is basically running non-stop. The manual compaction should be done some time tomorrow, so when totally caught up I will try again. What changes can be hoped for in 1470 or 1882 in terms of isolating compactions (or writes) affects on read requests?<br>
<br>Thanks<br><br><br><br><div class="gmail_quote">On Sat, Dec 18, 2010 at 2:36 PM, Peter Schuller <span dir="ltr">&lt;<a href="mailto:peter.schuller@infidyne.com">peter.schuller@infidyne.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="im">&gt; You are absolutely back to my main concern. Initially we were consistently<br>
&gt; seeing &lt; 10ms read latency and now we see 25ms (30GB sstable file), 50ms<br>
&gt; (100GB sstable file) and 65ms (330GB table file) read times for a single<br>
&gt; read with nothing else going on in the cluster. Concurrency is not our<br>
&gt; problem/concern (at this point), our problem is slow reads in total<br>
&gt; isolation. Frankly the concern is that a 2TB node with a 1TB sstable (worst<br>
&gt; case scenario) will result in &gt; 100ms read latency in total isolation.<br>
<br>
</div>So if you have a single non-concurrent client, along, submitting these<br>
reads that take 65 ms - are you disk bound (according to the last<br>
column of iostat -x 1), and how many reads per second (rps column) are<br>
you seeing relative to client reads? Is the number of disk reads per<br>
client read consistent with the actual number of sstables at the time?<br>
<br>
The behavior you're describing really does seem indicative of a<br>
problem, unless the the bottleneck is legitimately reads from disk<br>
from multiple sstables resulting from rows being spread over said<br>
sstables.<br>
<br>
--<br>
<font color="#888888">/ Peter Schuller<br>
</font></blockquote></div><br>
</blockquote></div><br></div></body></html>
--Apple-Mail-1-885161600--