cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Jordan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6840) 2.0.{5,6} node throws EOFExceptions on the Row mutation forwarding path during rolling upgrade from 1.2.15.
Date Fri, 21 Mar 2014 21:50:45 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13943603#comment-13943603
] 

Jeremiah Jordan commented on CASSANDRA-6840:
--------------------------------------------

Adding some follow up here for anyone hitting this issue.  Cross DC forwarding from C* 2.0.0-2.0.6
to C* 1.2.X is broken.  If you have multiple DC's and you already pushed through an upgrade,
you will want to run repair to make sure everything is in sync.  Hinted Handoff should take
care of the messed up forwards, but better safe than sorry, so I would run a repair.

> 2.0.{5,6} node throws EOFExceptions on the Row mutation forwarding path during rolling
upgrade from 1.2.15.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-6840
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6840
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Federico Piccinini
>            Assignee: Marcus Eriksson
>             Fix For: 2.0.7
>
>         Attachments: 0001-Read-id-properly-from-older-versions.patch
>
>
> During a rolling upgrade from 1.2.15 to 2.0.5 nodes running on 2.0.5 throw an EOFException:
> {noformat}
> ERROR [MutationStage:12] 2014-03-12 09:46:35,706 RowMutationVerbHandler.java (line 63)
Error in row mutation
> java.io.EOFException
> 	at java.io.DataInputStream.readFully(DataInputStream.java:197)
> 	at org.apache.cassandra.net.CompactEndpointSerializationHelper.deserialize(CompactEndpointSerializationHelper.java:37)
> 	at org.apache.cassandra.db.RowMutationVerbHandler.forwardToLocalNodes(RowMutationVerbHandler.java:81)
> 	at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:49)
> 	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> {noformat}
> In this specific context we have a setup with 3 datacenters, 3 nodes in each datacenter,
NetworkTopologyStrategy as placement_strategy with 3 replicas in each DC. We noticed the issue
on the only 2.0.5 node in the ring. All nodes run on Java7. We have tried to upgrade the node
on 2.0.5 to 2.0.6 but that didn't solve the issue.
> At a first glance it seems that the size of the size of the list of forward addresses
in  org.apache.cassandra.db.RowMutationVerbHandler.forwardToLocalNodes() in inconsistent with
the length of the InputStream, which causes the deserializer to try and read after the end
of the InputStream.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message