Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 16832 invoked from network); 5 Dec 2009 00:08:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Dec 2009 00:08:30 -0000 Received: (qmail 69159 invoked by uid 500); 5 Dec 2009 00:08:29 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 69137 invoked by uid 500); 5 Dec 2009 00:08:29 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 69127 invoked by uid 99); 5 Dec 2009 00:08:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Dec 2009 00:08:29 +0000 X-ASF-Spam-Status: No, hits=-2.6 required=5.0 tests=AWL,BAYES_00 X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bburruss@real.com designates 207.188.23.7 as permitted sender) Received: from [207.188.23.7] (HELO cir-el.real.com) (207.188.23.7) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Dec 2009 00:08:27 +0000 Received: from [172.21.131.225] ([::ffff:172.21.131.225]) (TLS: TLSv1/SSLv3,256bits,AES256-SHA) by cir-el.real.com with esmtp; Fri, 04 Dec 2009 16:08:07 -0800 id 001FC15F.4B19A467.0000082A Subject: Re: Persistently increasing read latency From: "B. Todd Burruss" To: cassandra-user@incubator.apache.org In-Reply-To: References: <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850190C6@GVW0432EXB.americas.hpqcorp.net> <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850C2317@GVW0432EXB.americas.hpqcorp.net> <59DD1BA8FD3C0F4C90771C18F2B5B53A4C85126629@GVW0432EXB.americas.hpqcorp.net> <1259953660.2898.96.camel@btoddb-laptop> <1259955614.2898.97.camel@btoddb-laptop> <1259965053.2898.194.camel@btoddb-laptop> Content-Type: text/plain; charset="UTF-8" Date: Fri, 04 Dec 2009 16:08:09 -0800 Message-ID: <1259971689.2898.201.camel@btoddb-laptop> Mime-Version: 1.0 X-Mailer: Evolution 2.28.1 Content-Transfer-Encoding: 7bit procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 5 1 29576 88400 27656 13797784 1 0 46 134 0 1 8 1 91 1 0 performance is definitely better now that caching is working, thx! however, the pending tasks in compaction pool are still not decreasing and i've reduced the load a bunch on the server. i'll be doing another test tonight with some modifications to my test client to simulate real world a bit better. On Fri, 2009-12-04 at 16:47 -0600, Jonathan Ellis wrote: > how much does vmstat say is being used for cache? > > if it's not most of the other 14, then read I/O doesn't sound like the culprit > > On Fri, Dec 4, 2009 at 4:17 PM, B. Todd Burruss wrote: > > thx for the tip. i've allocated -Xmx2G for cassandra out of 16G. > > > > > > On Fri, 2009-12-04 at 14:22 -0600, Brandon Williams wrote: > >> On Fri, Dec 4, 2009 at 1:40 PM, B. Todd Burruss > >> wrote: > >> i've actually allocatted plenty of RAM, but it isn't fully > >> used. the > >> read performance dwindles well before reaching a RAM limit. > >> > >> > >> You don't want to over-allocate memory to Cassandra, you want to leave > >> as much free as you can so the OS can cache the reads for you. > >> > >> > >> -Brandon > >> > > > > > >