Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of btalbot@aeriagames.com
 designates 74.125.149.246 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <B63551B57E91644AA79F2E09B2A7D042091A61B4AB@PROS-EMAIL.prosrm.com>
References: <B63551B57E91644AA79F2E09B2A7D042091A61B4AB@PROS-EMAIL.prosrm.com>
Date: Mon, 5 Nov 2012 11:16:40 -0800
Message-ID: 
 <CAPmnz8FTM89K3atcCD-WhQNVHPRL8-5vMyMvawPO2G2TFRZVyA@mail.gmail.com>
Subject: Re: How to upgrade a ring (0.8.9 nodes) to 1.1.5 with the minimal
 downtime?
From: Bryan Talbot <btalbot@aeriagames.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=047d7bae479236fddf04cdc4532d

--047d7bae479236fddf04cdc4532d
Content-Type: text/plain; charset=UTF-8

Do a rolling upgrade of the ring to 1.0.12 first and then upgrade to 1.1.x.
 After each rolling upgrade, you should probably do the recommend "nodetool
upgradesstables", etc.  The datastax documentation about upgrading might be
helpful for you: http://www.datastax.com/docs/1.1/install/upgrading

-Bryan


On Mon, Nov 5, 2012 at 10:55 AM, Yan Wu <ywu@prospricing.com> wrote:

> Hello,
>
> I have a Cassandra ring with 4 nodes in 0.8.9 and like to upgrade all
> nodes to 1.1.5.
> It would be great that the upgrade has no downtime or minimal downtime of
> the ring.
> After I brought down one of the nodes and upgraded it to 1.1.5, when I
> tried to bring it up,
> the new 1.1.5 node looks good but the rest of three 0.8.9 nodes started
> throwing exceptions:
> ---------------
> Fatal exception in thread Thread[GossipStage:2,5,main]
> java.lang.UnsupportedOperationException: Not a time-based UUID
> at
> org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:92)
>         at
> org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:75)
>         at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:707)
>         at
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:750)
>         at
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:809)
>         at
> org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ----------------
> Then later
> ----------------
> ERROR 12:03:20,925 Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach
> schema agreement with /xx.xx.xx.xx in 60000ms
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement
> with /xx.xx.xx.xx in 60000ms
>         at
> org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:293)
>         at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:304)
>         at
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
>         at
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:397)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         ... 3 more
> ----------------
>
> Any suggestions?   Thanks in advance.
>
> Yan
>

--047d7bae479236fddf04cdc4532d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Do a rolling upgrade of the ring to 1.0.12 first and then upgrade to 1.1.x.=
 =C2=A0After each rolling upgrade, you should probably do the recommend &qu=
ot;nodetool upgradesstables&quot;, etc. =C2=A0The datastax documentation ab=
out upgrading might be helpful for you:=C2=A0<a href=3D"http://www.datastax=
.com/docs/1.1/install/upgrading">http://www.datastax.com/docs/1.1/install/u=
pgrading</a><div>
<br></div><div><div>-Bryan</div><div><br><br><div class=3D"gmail_quote">On =
Mon, Nov 5, 2012 at 10:55 AM, Yan Wu <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:ywu@prospricing.com" target=3D"_blank">ywu@prospricing.com</a>&gt;</span>=
 wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hello,<br>
<br>
I have a Cassandra ring with 4 nodes in 0.8.9 and like to upgrade all nodes=
 to 1.1.5.<br>
It would be great that the upgrade has no downtime or minimal downtime of t=
he ring.<br>
After I brought down one of the nodes and upgraded it to 1.1.5, when I trie=
d to bring it up,<br>
the new 1.1.5 node looks good but the rest of three 0.8.9 nodes started thr=
owing exceptions:<br>
---------------<br>
Fatal exception in thread Thread[GossipStage:2,5,main]<br>
java.lang.UnsupportedOperationException: Not a time-based UUID<br>
at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.j=
ava:92)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.service.MigrationManage=
r.onAlive(MigrationManager.java:75)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.gms.Gossiper.markAlive(=
Gossiper.java:707)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.gms.Gossiper.handleMajo=
rStateChange(Gossiper.java:750)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.gms.Gossiper.applyState=
Locally(Gossiper.java:809)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.gms.GossipDigestAckVerb=
Handler.doVerb(GossipDigestAckVerbHandler.java:68)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.net.MessageDeliveryTask=
.run(MessageDeliveryTask.java:59)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.ThreadPoolExecutor$Work=
er.runTask(ThreadPoolExecutor.java:886)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.ThreadPoolExecutor$Work=
er.run(ThreadPoolExecutor.java:908)<br>
----------------<br>
Then later<br>
----------------<br>
ERROR 12:03:20,925 Fatal exception in thread Thread[HintedHandoff:1,1,main]=
<br>
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach sch=
ema agreement with /xx.xx.xx.xx in 60000ms<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.utils.WrappedRunnable.r=
un(WrappedRunnable.java:34)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.ThreadPoolExecutor$Work=
er.runTask(ThreadPoolExecutor.java:886)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.ThreadPoolExecutor$Work=
er.run(ThreadPoolExecutor.java:908)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.run(Thread.java:662)<br>
Caused by: java.lang.RuntimeException: Could not reach schema agreement wit=
h /xx.xx.xx.xx in 60000ms<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.db.HintedHandOffManager=
.waitForSchemaAgreement(HintedHandOffManager.java:293)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.db.HintedHandOffManager=
.deliverHintsToEndpoint(HintedHandOffManager.java:304)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.db.HintedHandOffManager=
.access$100(HintedHandOffManager.java:89)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.db.HintedHandOffManager=
$2.runMayThrow(HintedHandOffManager.java:397)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.utils.WrappedRunnable.r=
un(WrappedRunnable.java:30)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ... 3 more<br>
----------------<br>
<br>
Any suggestions? =C2=A0 Thanks in advance.<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
Yan<br>
</font></span></blockquote></div><br><br>
</div></div>

--047d7bae479236fddf04cdc4532d--