Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 47805 invoked from network); 1 Jun 2010 01:06:39 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 1 Jun 2010 01:06:39 -0000 Received: (qmail 19093 invoked by uid 500); 1 Jun 2010 01:06:38 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 19065 invoked by uid 500); 1 Jun 2010 01:06:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 19057 invoked by uid 99); 1 Jun 2010 01:06:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 01:06:37 +0000 X-ASF-Spam-Status: No, hits=-0.5 required=10.0 tests=AWL,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of stuhood@mailtrust.com designates 207.97.245.191 as permitted sender) Received: from [207.97.245.191] (HELO smtp191.iad.emailsrvr.com) (207.97.245.191) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 01:06:32 +0000 Received: from relay19.relay.iad.mlsrvr.com (localhost [127.0.0.1]) by relay19.relay.iad.mlsrvr.com (SMTP Server) with ESMTP id 8B0241B5370 for ; Mon, 31 May 2010 21:06:11 -0400 (EDT) Received: from dynamic2.wm-web.iad.mlsrvr.com (dynamic2.wm-web.iad.mlsrvr.com [192.168.2.151]) by relay19.relay.iad.mlsrvr.com (SMTP Server) with ESMTP id 86AE81B409A for ; Mon, 31 May 2010 21:06:11 -0400 (EDT) Received: from mailtrust.com (localhost [127.0.0.1]) by dynamic2.wm-web.iad.mlsrvr.com (Postfix) with ESMTP id 6F54728E8066 for ; Mon, 31 May 2010 21:06:11 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: stuhood@mailtrust.com, from: stu.hood@rackspace.com) with HTTP; Mon, 31 May 2010 20:06:11 -0500 (CDT) Date: Mon, 31 May 2010 20:06:11 -0500 (CDT) Subject: =?UTF-8?Q?Re:=20Inconsistency=20even=20after=20nodetool=20repair=3F?= From: "Stu Hood" To: user@cassandra.apache.org MIME-Version: 1.0 Content-Type: text/plain;charset=UTF-8 Content-Transfer-Encoding: quoted-printable Importance: Normal X-Priority: 3 (Normal) X-Type: plain In-Reply-To: References: <1275330599.954511335@192.168.2.228> Message-ID: <1275354371.4544600@192.168.2.231> X-Mailer: webmail8 The nodes only begin sending the data after the compactions have finished..= . https://issues.apache.org/jira/browse/CASSANDRA-579=0A=0A-----Original Me= ssage-----=0AFrom: "James Golick" =0ASent: Monday, M= ay 31, 2010 3:33pm=0ATo: user@cassandra.apache.org=0ASubject: Re: Inconsist= ency even after nodetool repair?=0A=0AIt may not have actually finished at = that point. Though, according to JMX,=0Aboth compactions of each CF had com= pleted, so I assumed it was done.=0A=0AOn Mon, May 31, 2010 at 11:29 AM, St= u Hood wrote:=0A=0A> Did you watch in the logs to = confirm that repair had actually finished? The=0A> `nodetool repair` call i= s not blocking before 0.6.3 (unreleased): see=0A> https://issues.apache.org= /jira/browse/CASSANDRA-1090=0A>=0A> -----Original Message-----=0A> From: "J= ames Golick" =0A> Sent: Sunday, May 30, 2010 3:43am= =0A> To: cassandra-user@incubator.apache.org=0A> Subject: Inconsistency eve= n after nodetool repair?=0A>=0A> I was getting a ton of these in my logs:= =0A>=0A> INFO [pool-1-thread-77] 2010-05-30 03:39:18,263 StorageProxy.java= (line=0A> 499) DigestMismatchException: Mismatch for key 142667/everything= =0A> (5546784193bbe4a4e066dc7fc142f589 vs 03edb87cd131a614d8256ddfdc50dd17)= =0A>=0A> The nodes were all getting really overloaded, presumably from the= =0A> read-repair, since everything else looked healthy. So, I ran nodetool= =0A> repair=0A> on both of our nodes, but even after running the repairs, I= 'm still seeing=0A> the same thing in the logs and high load on the nodes.= =0A>=0A> Does that sound right?=0A>=0A>=0A>=0A