cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Lerer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11393) dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
Date Tue, 05 Apr 2016 14:39:25 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226377#comment-15226377
] 

Benjamin Lerer commented on CASSANDRA-11393:
--------------------------------------------

I believe based on the different stack traces that we have in reality 2 different scenarios:
# In the case where the assertion is thrown by the {{LegacyReadCommandSerializer}} the problem
is caused by the fact that the coordinator though at the time where the message was created
that the replica was on version 2.1 and that it discover before serializing the message that
the replica has been upgraded to version 3.0.  
# In the case where the assertion is thrown by the {{ReadCommandSerializer}} the problem is
caused by the fact that the coordinator though at the time where the message was created that
the replica was on version 3.0 and that it discover before serializing the message that the
replica is in fact a 2.1 node. My guess is that this case could happen if the coordinator
has just restarted and that the message creation is performed just after the endPoint has
been added and before the version is set (MessageService returns the current version if it
has no version associated to the end point). 

> dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-11393
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11393
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Philip Thompson
>            Assignee: Benjamin Lerer
>              Labels: dtest
>
> We are seeing a failure in the upgrade tests that go from 2.1 to 3.0
> {code}
> node2: ERROR [SharedPool-Worker-2] 2016-03-10 20:05:17,865 Message.java:611 - Unexpected
exception during request; channel = [id: 0xeb79b477, /127.0.0.1:39613 => /127.0.0.2:9042]
> java.lang.AssertionError: null
> 	at org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1208)
~[main/:na]
> 	at org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1155)
~[main/:na]
> 	at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166) ~[main/:na]
> 	at org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
~[main/:na]
> 	at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
~[main/:na]
> 	at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758) ~[main/:na]
> 	at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701) ~[main/:na]
> 	at org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
~[main/:na]
> 	at org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
~[main/:na]
> 	at org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
~[main/:na]
> 	at org.apache.cassandra.service.AbstractReadExecutor$AlwaysSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:330)
~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654) ~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601) ~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520) ~[main/:na]
> 	at org.apache.cassandra.db.SinglePartitionReadCommand.execute(SinglePartitionReadCommand.java:302)
~[main/:na]
> 	at org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:67)
~[main/:na]
> 	at org.apache.cassandra.service.pager.SinglePartitionPager.fetchPage(SinglePartitionPager.java:34)
~[main/:na]
> 	at org.apache.cassandra.cql3.statements.SelectStatement$Pager$NormalPager.fetchPage(SelectStatement.java:297)
~[main/:na]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:333)
~[main/:na]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
~[main/:na]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76)
~[main/:na]
> 	at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
~[main/:na]
> 	at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:472)
~[main/:na]
> 	at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:449)
~[main/:na]
> 	at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
~[main/:na]
> 	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
[main/:na]
> 	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
[main/:na]
> 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[main/:na]
> 	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> {code}
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/24/testReport/upgrade_tests.upgrade_through_versions_test/ProtoV3Upgrade_2_1_UpTo_3_0_HEAD/rolling_upgrade_test
> Failed on CassCI build upgrade_tests-all #24
> The stack trace and context match that of CASSANDRA-10122. It looks like it may be the
same issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message