From user-return-26586-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Jun 1 01:43:32 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 796DF9BD9 for ; Fri, 1 Jun 2012 01:43:32 +0000 (UTC) Received: (qmail 9777 invoked by uid 500); 1 Jun 2012 01:43:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 9751 invoked by uid 500); 1 Jun 2012 01:43:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 9743 invoked by uid 99); 1 Jun 2012 01:43:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Jun 2012 01:43:30 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a80.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Jun 2012 01:43:22 +0000 Received: from homiemail-a80.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a80.g.dreamhost.com (Postfix) with ESMTP id 392BA37A06B for ; Thu, 31 May 2012 18:43:00 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=UbysRH9F7a KS5NleyDdTAW5ZaBXINgpJC+klnUCg4fdVuozez8gc/956Zuv3cQFTJF9YqNPDa/ K20u3ujhRqvVdNius2g/UoDSpXb5hkt91TrqynnPHIdM3uPOuf6BmMcwkiRV+GNC Uh3SDZ+qcmjCt+Owd04SVuH1C8i48j+Bc= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=J+7XQiaebou2zuOa 50wWNcr1bfE=; b=sOz43iiGg7+EsmotJEuhr6LaU9oBXwyfIjIlyyz+nE5Hx8sH jFEbfAAzE9HHiJEJFdm078Ql/V3l2cRw6/ZpyM4OMjPg9Uy3u2TfCDinH+usJADl IbkxE59DIQj1w+Fivw4i6U7rAqk6eeS2R3Nw9GTKHX8ovne8tb3W5uXLHuQ= Received: from [192.168.2.189] (unknown [116.90.132.105]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a80.g.dreamhost.com (Postfix) with ESMTPSA id BD56E37A065 for ; Thu, 31 May 2012 18:42:59 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: multipart/alternative; boundary="Apple-Mail=_74D178A7-8238-4D6C-AB16-01753EF1A766" Subject: Re: Invalid Counter Shard errors? Date: Fri, 1 Jun 2012 13:42:57 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: X-Mailer: Apple Mail (2.1278) --Apple-Mail=_74D178A7-8238-4D6C-AB16-01753EF1A766 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 I suggest creating a ticket on = https://issues.apache.org/jira/browse/CASSANDRA with the details. If it is an immediate concern see if you can find someone in the = #cassandra chat room http://cassandra.apache.org/ Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/06/2012, at 3:20 AM, Charles Brophy wrote: > Hi guys, >=20 > We're running a three node cluster of cassandra 1.1 servers, = originally 1.0.7 and immediately after the upgrade the error logs of all = three servers began filling up with the following message: >=20 > ERROR [ReplicateOnWriteStage:177] 2012-05-31 08:17:02,236 = CounterContext.java (line 381) invalid counter shard detected; = (3438afc0-7e71-11e1-0000-da5a9d01e7f7, 3, 4) and = (3438afc0-7e71-11e1-0000-da5a9d01e7f7, 3, 7) differ only in count; will = pick highest to self-heal; this indicates a bug or corruption generated = a bad counter shard >=20 > ERROR [ValidationExecutor:20] 2012-05-31 08:17:01,570 = CounterContext.java (line 381) invalid counter shard detected; = (343cf580-7e71-11e1-0000-ebc411012bff, 14, 27) and = (343cf580-7e71-11e1-0000-ebc411012bff, 14, 21) differ only in count; = will pick highest to self-heal; this indicates a bug or corruption = generated a bad counter shard >=20 > The counts change but the errors are constant. What is the best course = of action? Google only turns up the source code for these errors. >=20 > Thanks! > Charles >=20 >=20 --Apple-Mail=_74D178A7-8238-4D6C-AB16-01753EF1A766 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 I = suggest creating a ticket on https://issues.ap= ache.org/jira/browse/CASSANDRA with the = details.

If it is an immediate concern see if you can = find someone in the #cassandra chat room http://cassandra.apache.org/

Cheers

http://www.thelastpickle.com

On 1/06/2012, at 3:20 AM, Charles Brophy wrote:

Hi = guys,

We're running a three node cluster of cassandra = 1.1 servers, originally 1.0.7 and immediately after the upgrade the = error logs of all three servers began filling up with the following = message:

ERROR [ReplicateOnWriteStage:177] 2012-05-31 = 08:17:02,236 CounterContext.java (line 381) invalid counter shard = detected; (3438afc0-7e71-11e1-0000-da5a9d01e7f7, 3, 4) and = (3438afc0-7e71-11e1-0000-da5a9d01e7f7, 3, 7) differ only in count; will = pick highest to self-heal; this indicates a bug or corruption generated = a bad counter shard

ERROR [ValidationExecutor:20] 2012-05-31 = 08:17:01,570 CounterContext.java (line 381) invalid counter shard = detected; (343cf580-7e71-11e1-0000-ebc411012bff, 14, 27) and = (343cf580-7e71-11e1-0000-ebc411012bff, 14, 21) differ only in count; = will pick highest to self-heal; this indicates a bug or corruption = generated a bad counter shard

The counts change but the errors are constant. What = is the best course of action? Google only turns up the source code for = these = errors.

Thanks!
Charles



= --Apple-Mail=_74D178A7-8238-4D6C-AB16-01753EF1A766--