Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C5642D3BB for ; Mon, 5 Nov 2012 18:55:43 +0000 (UTC) Received: (qmail 38070 invoked by uid 500); 5 Nov 2012 18:55:41 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 38040 invoked by uid 500); 5 Nov 2012 18:55:41 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 38032 invoked by uid 99); 5 Nov 2012 18:55:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Nov 2012 18:55:41 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ywu@prospricing.com designates 64.18.2.28 as permitted sender) Received: from [64.18.2.28] (HELO exprod7og125.obsmtp.com) (64.18.2.28) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 05 Nov 2012 18:55:32 +0000 Received: from pros-mail4.prosrm.com ([209.133.111.65]) by exprod7ob125.postini.com ([64.18.6.12]) with SMTP ID DSNKUJgLji3yXtVIIH3iMVIWUqa4t9MVq1/M@postini.com; Mon, 05 Nov 2012 10:55:11 PST Received: from PROS-EMAIL.prosrm.com ([172.16.1.239]) by pros-mail4.prosrm.com ([172.16.1.218]) with mapi; Mon, 5 Nov 2012 12:55:11 -0600 From: Yan Wu To: "user@cassandra.apache.org" Date: Mon, 5 Nov 2012 12:55:11 -0600 Subject: How to upgrade a ring (0.8.9 nodes) to 1.1.5 with the minimal downtime? Thread-Topic: How to upgrade a ring (0.8.9 nodes) to 1.1.5 with the minimal downtime? Thread-Index: Ac27hxE5EcoR2aZ9SMq3Ei3ycbA+iQ== Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Hello,=20 I have a Cassandra ring with 4 nodes in 0.8.9 and like to upgrade all nodes= to 1.1.5. It would be great that the upgrade has no downtime or minimal downtime of t= he ring. After I brought down one of the nodes and upgraded it to 1.1.5, when I trie= d to bring it up,=20 the new 1.1.5 node looks good but the rest of three 0.8.9 nodes started thr= owing exceptions: --------------- Fatal exception in thread Thread[GossipStage:2,5,main] java.lang.UnsupportedOperationException: Not a time-based UUID at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.j= ava:92) at org.apache.cassandra.service.MigrationManager.onAlive(MigrationM= anager.java:75) at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:707) at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossipe= r.java:750) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.jav= a:809) at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(Gossi= pDigestAckVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDelivery= Task.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoo= lExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:908) ---------------- Then later ---------------- ERROR 12:03:20,925 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.lang.RuntimeException: Could not reach sch= ema agreement with /xx.xx.xx.xx in 60000ms at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.j= ava:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoo= lExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: Could not reach schema agreement wit= h /xx.xx.xx.xx in 60000ms at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreem= ent(HintedHandOffManager.java:293) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpo= int(HintedHandOffManager.java:304) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHa= ndOffManager.java:89) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(Hinte= dHandOffManager.java:397) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.j= ava:30) ... 3 more ---------------- Any suggestions? Thanks in advance. Yan