Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 30D016F31 for ; Tue, 14 Jun 2011 08:26:03 +0000 (UTC) Received: (qmail 73195 invoked by uid 500); 14 Jun 2011 08:26:01 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 73165 invoked by uid 500); 14 Jun 2011 08:26:00 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 73157 invoked by uid 99); 14 Jun 2011 08:26:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 08:26:00 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sdolgy@gmail.com designates 209.85.212.44 as permitted sender) Received: from [209.85.212.44] (HELO mail-vw0-f44.google.com) (209.85.212.44) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 08:25:56 +0000 Received: by vws12 with SMTP id 12so6234537vws.31 for ; Tue, 14 Jun 2011 01:25:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=CFe9f1/W2XaeGe/mhZWgceG59kSjXpDltU0rHtdiG4o=; b=nk1o+B/zVKY4m900iVv65vj68ViMj5+rKXyh4kXUoA4WSQHbDbeixv2xb5bBIG7sKR KbAHFr8g8ZQrtAU316HFB4EUw2/rohRgF7aBJ7+dKku1CDX98UPOmAkLQAoi98mfk4NQ 5NT9hoy+VyXAgk1+sq8syXN8aVYZdsXe9KDWE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=Nr9cG2YKEx2nJ97ONm4B7p6bbf73Vo3DZHkMLQ7W2G8HNeQ/0+t98wetwjECjIZIur XK+3gba94Rp7aIjnc/Ax1eL/KhSDE1lkMf7G7tMdKEdiCXa6E/qZeGk1kRD+SQfUx0w5 tAV9ZUmG+KIFWGXcLu7Jms88OKyVpLlxfj2xA= MIME-Version: 1.0 Received: by 10.52.160.99 with SMTP id xj3mr5472763vdb.261.1308039933658; Tue, 14 Jun 2011 01:25:33 -0700 (PDT) Received: by 10.52.162.69 with HTTP; Tue, 14 Jun 2011 01:25:33 -0700 (PDT) Received: by 10.52.162.69 with HTTP; Tue, 14 Jun 2011 01:25:33 -0700 (PDT) In-Reply-To: References: Date: Tue, 14 Jun 2011 10:25:33 +0200 Message-ID: Subject: Re: odd logs after repair From: Sasha Dolgy To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=bcaec53f917999071c04a5a7c77f --bcaec53f917999071c04a5a7c77f Content-Type: text/plain; charset=ISO-8859-1 Hi Sylvain, I verified on all nodes with nodetool version that they are 0.8 and have even restarted nodes. Still persists. The four nodes all report similar errors about the other nodes. When i upgraded to 0.8 maybe there were relics about the keyspace that say it's from an earlier version? I need to create a new keyspace to see if that fixes the error.... On Jun 14, 2011 10:08 AM, "Sylvain Lebresne" wrote: > The exception itself is a bug (I've created > https://issues.apache.org/jira/browse/CASSANDRA-2767 to fix it). > > However, the important message is the previous one (Even if the > exception was not thrown, repair wouldn't be able to work correctly, > so the fact that the exception is thrown is not such a big deal). > Apparently, from the standpoint of whomever node this logs is from, > the node 10.128.34.18 is still running 0.7. You should check if it is > the case (restarting 10.128.34.18 and look for something like > 'Cassandra version: 0.8.0' is one solution). If the does does run > 0.8.0 and you still get this error, then it would point to a problem > with our detection of the nodes. > > -- > Sylvain > > On Tue, Jun 14, 2011 at 9:55 AM, Sasha Dolgy wrote: >> Hi ... >> >> Does anyone else see these type of INFO messages in their log files, >> or is i just me..? >> >> INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 >> 21:28:39,877 AntiEntropyService.java (line 177) Excluding >> /10.128.34.18 from repair because it is on version 0.7 or sooner. You >> should consider updating this node before running repair again. >> ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 >> 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception >> in thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI >> Runtime] >> java.util.ConcurrentModificationException >> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) >> at java.util.HashMap$KeyIterator.next(HashMap.java:828) >> at org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173) >> at org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776) >> >> I'm at a loss as to why this is showing up in the logs. >> -sd >> >> On Mon, Jun 13, 2011 at 3:58 PM, Sasha Dolgy wrote: >>> hm. that's not it. we've been using a non-standard jmx port for some time.... >>> >>> i've dropped the keyspace and recreated ... >>> >>> wonder if that'll help >>> >>> On Mon, Jun 13, 2011 at 3:57 PM, Tyler Hobbs wrote: >>>> On Mon, Jun 13, 2011 at 8:41 AM, Sasha Dolgy wrote: >>>>> >>>>> I recall there being a discussion about a default port changing from >>>>> 0.7.x to 0.8.x ...this was JMX, correct? Or were there others. >>>> >>>> Yes, the default JMX port changed from 8080 to 7199. I don't think there >>>> were any others. >> --bcaec53f917999071c04a5a7c77f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Hi Sylvain,

I verified on all nodes with nodetool version that they are 0.8 and have= even restarted nodes.=A0 Still persists.=A0 The four nodes all report simi= lar errors about the other nodes.

When i upgraded to 0.8 maybe there were relics about the keyspace that s= ay it's from an earlier version?=A0

I need to create a new keyspace to see if that fixes the error....

On Jun 14, 2011 10:08 AM, "Sylvain Lebresne= " <sylvain@datastax.com= > wrote:
> The exception itself is a bug (I&#= 39;ve created
> https= ://issues.apache.org/jira/browse/CASSANDRA-2767 to fix it).
> > However, the important message is the previous one (Even if the
> exception was not thrown, repair wouldn't be able to work correctl= y,
> so the fact that the exception is thrown is not such a big deal)= .
> Apparently, from the standpoint of whomever node this logs is fro= m,
> the node 10.128.34.18 is still running 0.7. You should check if it is<= br>> the case (restarting 10.128.34.18 and look for something like
&g= t; 'Cassandra version: 0.8.0' is one solution). If the does does ru= n
> 0.8.0 and you still get this error, then it would point to a problem> with our detection of the nodes.
>
> --
> Sylvain=
>
> On Tue, Jun 14, 2011 at 9:55 AM, Sasha Dolgy <sdolgy@gmail.com> wrote:
>> Hi ...
>>
>> Does anyone else see these type of = INFO messages in their log files,
>> or is i just me..?
>>= ;
>> INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 201= 1-06-13
>> 21:28:39,877 AntiEntropyService.java (line 177) Excluding
>&= gt; /10.128.34.18 from repair because i= t is on version 0.7 or sooner. You
>> should consider updating thi= s node before running repair again.
>> ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06= -13
>> 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal = exception
>> in thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f= 6-f1464ec8bdec,5,RMI
>> Runtime]
>> java.util.ConcurrentModificationException
= >> =A0 =A0 =A0 at java.util.HashMap$HashIterator.nextEntry(HashMap.ja= va:793)
>> =A0 =A0 =A0 at java.util.HashMap$KeyIterator.next(HashM= ap.java:828)
>> =A0 =A0 =A0 at org.apache.cassandra.service.AntiEntropyService.get= Neighbors(AntiEntropyService.java:173)
>> =A0 =A0 =A0 at org.apach= e.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService= .java:776)
>>
>> I'm at a loss as to why this is showing up in the = logs.
>> -sd
>>
>> On Mon, Jun 13, 2011 at 3:58 = PM, Sasha Dolgy <sdolgy@gmail.com> wrote:
>>> hm. =A0that's not it. =A0we've been using a non-standa= rd jmx port for some time....
>>>
>>> i've drop= ped the keyspace and recreated ...
>>>
>>> wonder i= f that'll help
>>>
>>> On Mon, Jun 13, 2011 at 3:57 PM, Tyler Hobbs &= lt;tyler@datastax.com> wrote:<= br>>>>> On Mon, Jun 13, 2011 at 8:41 AM, Sasha Dolgy <sdolgy@gmail.com> wrote:
>>>>>
>>>>> I recall there being a discuss= ion about a default port changing from
>>>>> 0.7.x to 0.8= .x ...this was JMX, correct? =A0Or were there others.
>>>> >>>> Yes, the default JMX port changed from 8080 to 7199.=A0 I = don't think there
>>>> were any others.
>>
<= /div> --bcaec53f917999071c04a5a7c77f--