Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 333816681 for ; Tue, 14 Jun 2011 20:57:10 +0000 (UTC) Received: (qmail 78201 invoked by uid 500); 14 Jun 2011 20:57:10 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 78179 invoked by uid 500); 14 Jun 2011 20:57:10 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 78171 invoked by uid 99); 14 Jun 2011 20:57:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 20:57:10 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 20:57:08 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 01330416FA9 for ; Tue, 14 Jun 2011 20:56:48 +0000 (UTC) Date: Tue, 14 Jun 2011 20:56:47 +0000 (UTC) From: "Sasha Dolgy (JIRA)" To: commits@cassandra.apache.org Message-ID: <475647068.4239.1308085008001.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <1279009063.2133.1308042773948.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Commented] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049411#comment-13049411 ] Sasha Dolgy commented on CASSANDRA-2768: ---------------------------------------- I can assure you they are all on 0.8.0 if the output of "nodetool version" is correct cassandra@ip-10-128-34-18:~$ nodetool -h 10.128.34.18 -p 9090 ring Address Status State Load Owns Token 170141183460469231731687303715884105726 10.128.103.148 Up Normal 1.02 MB 11.22% 19095547144942516281182777765338228798 10.128.94.227 Up Normal 667.56 KB 22.11% 56713727820156410577229101238628035242 10.128.34.18 Up Normal 688.1 KB 33.33% 113427455640312821154458202477256070484 10.128.90.109 Up Normal 1.11 MB 33.33% 170141183460469231731687303715884105726 cassandra@ip-10-128-34-18:~$ cassandra@ip-10-128-34-18:~$ nodetool -h 10.128.34.18 -p 9090 version ReleaseVersion: 0.8.0 cassandra@ip-10-128-34-18:~$ cassandra@ip-10-128-94-227:~$ nodetool -h 10.128.94.227 -p 9090 version ReleaseVersion: 0.8.0 cassandra@ip-10-128-94-227:~$ cassandra@ip-10-128-90-109:~$ nodetool -h 10.128.90.109 -p 9090 version ReleaseVersion: 0.8.0 cassandra@ip-10-128-90-109:~$ cassandra@ip-10-128-103-148:~$ nodetool -h 10.128.103.148 -p 9090 version ReleaseVersion: 0.8.0 cassandra@ip-10-128-103-148:~$ > AntiEntropyService excluding nodes that are on version 0.7 or sooner > -------------------------------------------------------------------- > > Key: CASSANDRA-2768 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2768 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 0.8.0 > Environment: 4 node environment -- > Originally 0.7.6-2 with a Keyspace defined with RF=3 > Upgraded all nodes ( 1 at a time ) to version 0.8.0: For each node, the node was shut down, new version was turned on, using the existing data files / directories and a nodetool repair was run. > Reporter: Sasha Dolgy > Assignee: Brandon Williams > > When I run nodetool repair on any of the nodes, the /var/log/cassandra/system.log reports errors similar to: > INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from repair because it is on version 0.7 or sooner. You should consider updating this node before running repair again. > ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI Runtime] > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) > at java.util.HashMap$KeyIterator.next(HashMap.java:828) > at org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173) > at org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776) > The INFO message and subsequent ERROR message are logged for 2 nodes .. I suspect that this is because RF=3. > nodetool ring shows that all nodes are up. > Client connections (read / write) are not having issues.. > nodetool version on all nodes shows that each node is 0.8.0 > At suggestion of some contributors, I have restarted each node and tried to run a nodetool repair again ... the result is the same with the messages being logged. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira