incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra 1.0.10 to 1.2.3 upgrade "post-mortem"
Date Thu, 04 Apr 2013 01:24:07 GMT
> I just wanted to share our experience of upgrading 1.0.10 to 1.2.3
In general it's dangerous to skip a major release when upgrading. 

> ERROR [MutationStage:33] 2013-03-31 09:00:02,899 CassandraDaemon.java (line 164) Exception
in thread Thread[MutationStage:33,5,main]
> java.lang.AssertionError: Missing host ID for 10.0.1.8
Was 10.0.1.8 been updated ?

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 3/04/2013, at 4:09 AM, Rustam Aliyev <rustam.lists@code.az> wrote:

> Hi,
> 
> I just wanted to share our experience of upgrading 1.0.10 to 1.2.3. It happened that
first we upgraded both of our two seeds to 1.2.3. And basically after that old nodes couldn't
communicate with new ones anymore. Cluster was down until we upgraded all nodes to 1.2.3.
We don't have many nodes and that process didn't took long. Yet it caused outage for ~10 mins.
> 
> Here are some logs:
> 
> On the new, freshly upgraded seed node (v1.2.3):
> ----------------
> ERROR [OptionalTasks:1] 2013-03-31 08:48:19,370 CassandraDaemon.java (line 164) Exception
in thread Thread[OptionalTasks:1,5,main]
> java.lang.NullPointerException
> at org.apache.cassandra.service.MigrationManager$1.run(MigrationManager.java:137)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> WARN [MutationStage:20] 2013-03-31 08:48:23,613 StorageProxy.java (line 577) Unable to
store hint for host with missing ID, /10.0.1.8 (old node?)
> 
> ....
> 
> ERROR [MutationStage:33] 2013-03-31 09:00:02,899 CassandraDaemon.java (line 164) Exception
in thread Thread[MutationStage:33,5,main]
> java.lang.AssertionError: Missing host ID for 10.0.1.8
> at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:580)
> at org.apache.cassandra.service.StorageProxy$5.runMayThrow(StorageProxy.java:555)
> at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1643)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> ----------------
> 
> At the same time, old nodes (v1.0.10) were blinded:
> 
> ----------------
> ERROR [RequestResponseStage:441] 2013-03-31 09:04:07,955 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[RequestResponseStage:441,5,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
> at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:132)
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:100)
> at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81)
> at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
> ... 6 more
> 
> .....
> 
> INFO [GossipStage:3] 2013-03-31 09:06:08,885 Gossiper.java (line 804) InetAddress /10.0.1.8
is now UP
> ERROR [GossipStage:3] 2013-03-31 09:06:08,885 AbstractCassandraDaemon.java (line 139)
Fatal exception in thread Thread[GossipStage:3,5,main]
> java.lang.UnsupportedOperationException: Not a time-based UUID
> at java.util.UUID.timestamp(UUID.java:308)
> at org.apache.cassandra.service.MigrationManager.updateHighestKnown(MigrationManager.java:121)
> at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:99)
> at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:83)
> at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:806)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:923)
> at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:3] 2013-03-31 09:06:08,886 AbstractCassandraDaemon.java (line 139)
Fatal exception in thread Thread[GossipStage:3,5,main]
> java.lang.UnsupportedOperationException: Not a time-based UUID
> at java.util.UUID.timestamp(UUID.java:308)
> at org.apache.cassandra.service.MigrationManager.updateHighestKnown(MigrationManager.java:121)
> at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:99)
> at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:83)
> at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:806)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:923)
> at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ----------
> 
> I've seen other people reporting similar problem with 1.1.x.
> 
> Not sure, but maybe we could avoid this by upgrading seeds last.
> 
> Best,
> Rustam.
> 


Mime
View raw message