From cassandra-user-return-1807-apmail-incubator-cassandra-user-archive=incubator.apache.org@incubator.apache.org Tue Dec 15 02:04:16 2009 Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 31773 invoked from network); 15 Dec 2009 02:04:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 15 Dec 2009 02:04:16 -0000 Received: (qmail 85879 invoked by uid 500); 15 Dec 2009 02:04:15 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 85838 invoked by uid 500); 15 Dec 2009 02:04:15 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 85829 invoked by uid 99); 15 Dec 2009 02:04:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Dec 2009 02:04:15 +0000 X-ASF-Spam-Status: No, hits=-2.6 required=5.0 tests=AWL,BAYES_00 X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bburruss@real.com designates 207.188.23.4 as permitted sender) Received: from [207.188.23.4] (HELO kal-el.real.com) (207.188.23.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Dec 2009 02:04:11 +0000 Received: from seacas01.corp.real.com ([::ffff:192.168.139.56]) (TLS: TLSv1/SSLv3,128bits,AES128-SHA) by kal-el.real.com with esmtp; Mon, 14 Dec 2009 18:03:50 -0800 id 00080033.4B26EE86.00002DD5 Received: from seambx.corp.real.com ([fe80::2d15:fda7:b3b8:e268]) by seacas01.corp.real.com ([192.168.139.56]) with mapi; Mon, 14 Dec 2009 18:03:50 -0800 From: Brian Burruss To: "cassandra-user@incubator.apache.org" Date: Mon, 14 Dec 2009 18:03:49 -0800 Subject: Re: read latency creaping up Thread-Topic: read latency creaping up Thread-Index: Acp9JUDmPAjohLbASUqy+iZtHvALfwABZaba Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 I plan to investigate that next. There isn't a straightforward "disconnect= " that I could easily see. Ian Holsman wrote: can you make it so that the client restarts the connection every 30m or so = ? It could be an issue in thrift or something with long-lived connections. On Dec 15, 2009, at 10:16 AM, Brian Burruss wrote: > i agree. i don't know anything about thrift, and i don't know how it kee= ps connections open or manages resources from a client or server perspectiv= e, but this situation suggests that maybe killing the clients is forcing th= e server to free something. > > how's that sound :) > > ________________________________________ > From: Jonathan Ellis [jbellis@gmail.com] > Sent: Monday, December 14, 2009 3:12 PM > To: cassandra-user@incubator.apache.org > Subject: Re: read latency creaping up > > hmm, me neither > > but, I can't think how restarting the client would, either :) > > On Mon, Dec 14, 2009 at 4:59 PM, Brian Burruss wrote: >> Well not sure how that would affect he latency as reported by the Cassan= dra server using nodeprobe cfstats >> >> Jonathan Ellis wrote: >> >> >> possibly the clients are running into memory pressure? >> >> On Mon, Dec 14, 2009 at 4:27 PM, Brian Burruss wrote= : >>> thx, i'm actually the "B. Todd Burruss" in that thread .. we changed o= ur email system and well now, i'm just Brian .. long story. >>> >>> anyway, in this case it isn't compaction pendings as i can kill the cli= ents and immediately restart and the latency is back to a reasonable number= . i'm still investigating. >>> >>> thx! >>> ________________________________________ >>> From: Eric Evans [eevans@rackspace.com] >>> Sent: Monday, December 14, 2009 8:23 AM >>> To: cassandra-user@incubator.apache.org >>> Subject: RE: read latency creaping up >>> >>> On Sun, 2009-12-13 at 13:18 -0800, Brian Burruss wrote: >>>> if this isn't a known issue, lemme do some more investigating. my >>>> test client becomes "more random" with reads as time progresses, so >>>> possibly this is what causes the latency issue. however, all that >>>> being said, the performance really becomes bad after a while. >>> >>> Have a look at the following thread: >>> >>> http://thread.gmane.org/gmane.comp.db.cassandra.user/1402 >>> >>> >>> -- >>> Eric Evans >>> eevans@rackspace.com >>> >>> >> -- Ian Holsman Ian@Holsman.net