Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 25696 invoked from network); 21 Jan 2010 21:49:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 21 Jan 2010 21:49:18 -0000 Received: (qmail 54530 invoked by uid 500); 21 Jan 2010 21:49:17 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 54496 invoked by uid 500); 21 Jan 2010 21:49:17 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 54487 invoked by uid 99); 21 Jan 2010 21:49:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jan 2010 21:49:17 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bburruss@real.com designates 207.188.23.4 as permitted sender) Received: from [207.188.23.4] (HELO kal-el.real.com) (207.188.23.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jan 2010 21:49:11 +0000 Received: from seacas02.corp.real.com ([::ffff:192.168.139.57]) (TLS: TLSv1/SSLv3,128bits,AES128-SHA) by kal-el.real.com with esmtp; Thu, 21 Jan 2010 13:48:50 -0800 id 0008003E.4B58CBC2.000020A9 Received: from [172.21.131.225] (192.168.198.6) by seacas02.corp.real.com (192.168.139.57) with Microsoft SMTP Server id 8.2.176.0; Thu, 21 Jan 2010 13:48:50 -0800 Subject: Re: load balancing From: "B. Todd Burruss" To: "cassandra-user@incubator.apache.org" In-Reply-To: References: <1264024749.2093.17.camel@btoddb-laptop> Content-Type: text/plain; charset="UTF-8" Date: Thu, 21 Jan 2010 13:48:51 -0800 Message-ID: <1264110531.32541.1.camel@btoddb-laptop> MIME-Version: 1.0 X-Mailer: Evolution 2.28.1 Content-Transfer-Encoding: 7bit Old-Return-Path: bburruss@real.com i do have the fix. just checked the code. On Wed, 2010-01-20 at 19:16 -0800, Jonathan Ellis wrote: > are you on a recent enough trunk to have the fix for > https://issues.apache.org/jira/browse/CASSANDRA-715 ? We had a > regression for a while that caused HH delivery to infinite loop > consuming a ton of CPU, which sounds like what you're seeing. > > On Wed, Jan 20, 2010 at 3:59 PM, B. Todd Burruss wrote: > > feedback on trunk code ... i'm using code from trunk, 4 node cluster, > > RF=3, W=Q, R=Q and did a nodeprobe loadbalance on the hot node. the > > cluster ran through the night and this morning i noticed a lot of > > pending hinted-handoff-pool via tpstats. looking thru the logs i notice > > that the nodes are very frequently "InetAddress /192.168.132.102 is now > > dead" then back UP .. flapping of the nodes. > > > > is this triggering the hinted-handoff and is there a way to alleviate > > this (other than lowering the load)? it seems that the hinted handdoffs > > have slowed or stopped the loadbalance. > > > > i've been trying to load up the cluster with data, so i'm trying to do > > 2000+ puts a second with little or no reads. it seems like the node > > that is loadbalancing is bringing the performance of the cluster way > > down. > > > > > >