incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shashilpi Krishan <Shashilpi.Kris...@wizecommerce.com>
Subject RE: Row Mutation Errors while upgrading to Cassandra2.0
Date Mon, 23 Sep 2013 08:47:41 GMT
Thanks for replying Sankalp

One has to start and test, and we have many such clusters and this cluster hold trivial data
and a candidate for the trial. But if others also are seeing problems with 2.0 then we won't
proceed to do that on our main clusters and let them upgrade to 1.2.9 only.

Before doing this in production we tested 2.0 successfully in test environment and at that
time, no such issues were noticed (perhaps testing happened in single DC is the reason for
that).

BTW, we are using GossipingPropertyFileSnitch ....so could that be the reason.
________________________________
Thanks & Regards

Shashilpi Krishan

From: sankalp kohli [mailto:kohlisankalp@gmail.com]
Sent: Monday, September 23, 2013 8:01 AM
To: user@cassandra.apache.org
Subject: Re: Row Mutation Errors while upgrading to Cassandra2.0

You are upgrading to 2.0 in Prod? What is the urgency?

On Sun, Sep 22, 2013 at 7:02 PM, Shashilpi Krishan <Shashilpi.Krishan@wizecommerce.com<mailto:Shashilpi.Krishan@wizecommerce.com>>
wrote:
Hi Everyone.

We had a Cassandra cluster (running with v1.0.7) spread across 3 data centers with each data
center having 16 nodes. We started upgrading that to 2.0 but realized that we can't go directly
to 2.0 due to read failures, hence to avoid down time we have to go from 1.0 --> 1.1 -->
1.2 --> 2.0.

Now problem is while upgrading from 1.2 --> 2.0 we saw below errors flooding the system.log
files in one data center only until we upgraded all the nodes in every DC to 2.0. We think
that moment last node was upgraded then this error was gone. Does anyone has idea what could
have been causing this? Is it because of some version incompatibility?

ERROR [MutationStage:208] 2013-09-21 17:51:08,163 RowMutationVerbHandler.java (line 63) Error
in row mutation
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:197)
        at org.apache.cassandra.net.CompactEndpointSerializationHelper.deserialize(CompactEndpointSerializationHelper.java:37)
        at org.apache.cassandra.db.RowMutationVerbHandler.forwardToLocalNodes(RowMutationVerbHandler.java:81)
        at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:49)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
________________________________
Thanks & Regards

Shashilpi Krishan


CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient, please contact the sender
by reply email and destroy all copies of the original message along with any attachments,
from your computer system. If you are the intended recipient, please be advised that the content
of this message is subject to access, review and disclosure by the sender's Email System Administrator.


CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient, please contact the sender
by reply email and destroy all copies of the original message along with any attachments,
from your computer system. If you are the intended recipient, please be advised that the content
of this message is subject to access, review and disclosure by the sender's Email System Administrator.

Mime
View raw message