Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0D27C8D1 for ; Mon, 23 Sep 2013 02:03:08 +0000 (UTC) Received: (qmail 48415 invoked by uid 500); 23 Sep 2013 02:03:06 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 48392 invoked by uid 500); 23 Sep 2013 02:03:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 48384 invoked by uid 99); 23 Sep 2013 02:03:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Sep 2013 02:03:06 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Shashilpi.Krishan@wizecommerce.com designates 207.46.163.244 as permitted sender) Received: from [207.46.163.244] (HELO na01-bl2-obe.outbound.protection.outlook.com) (207.46.163.244) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Sep 2013 02:02:58 +0000 Received: from BN1PR04MB422.namprd04.prod.outlook.com (10.141.58.150) by BN1PR04MB421.namprd04.prod.outlook.com (10.141.58.143) with Microsoft SMTP Server (TLS) id 15.0.775.9; Mon, 23 Sep 2013 02:02:22 +0000 Received: from BN1PR04MB422.namprd04.prod.outlook.com ([169.254.9.98]) by BN1PR04MB422.namprd04.prod.outlook.com ([169.254.9.98]) with mapi id 15.00.0775.005; Mon, 23 Sep 2013 02:02:21 +0000 From: Shashilpi Krishan To: "user@cassandra.apache.org" Subject: Row Mutation Errors while upgrading to Cassandra2.0 Thread-Topic: Row Mutation Errors while upgrading to Cassandra2.0 Thread-Index: Ac63/562lorQIgoxSRuTRO3txaqAXA== Date: Mon, 23 Sep 2013 02:02:21 +0000 Message-ID: <4a3240330db04318b06536d0e83d7652@BN1PR04MB422.namprd04.prod.outlook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [115.113.53.206] x-forefront-prvs: 09781D4C35 x-forefront-antispam-report: SFV:NSPM;SFS:(377424004)(53754006)(61754003)(199002)(189002)(51856001)(16236675002)(46102001)(79102001)(77982001)(59766001)(56776001)(53806001)(54316002)(47736001)(74662001)(47976001)(50986001)(49866001)(47446002)(74316001)(74502001)(76482001)(4396001)(81342001)(31966008)(81542001)(69226001)(74706001)(77096001)(56816003)(76576001)(33646001)(81816001)(81686001)(76176001)(80022001)(65816001)(80976001)(83322001)(83072001)(66066001)(63696002)(76786001)(54356001)(74876001)(19580395003)(76796001)(74366001)(567094001)(24736002);DIR:OUT;SFP:;SCL:1;SRVR:BN1PR04MB421;H:BN1PR04MB422.namprd04.prod.outlook.com;CLIP:115.113.53.206;FPR:;RD:InfoNoRecords;A:1;MX:1;LANG:en; Content-Type: multipart/alternative; boundary="_000_4a3240330db04318b06536d0e83d7652BN1PR04MB422namprd04pro_" MIME-Version: 1.0 X-OriginatorOrg: wizecommerce.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_4a3240330db04318b06536d0e83d7652BN1PR04MB422namprd04pro_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Everyone. We had a Cassandra cluster (running with v1.0.7) spread across 3 data cente= rs with each data center having 16 nodes. We started upgrading that to 2.0 = but realized that we can't go directly to 2.0 due to read failures, hence t= o avoid down time we have to go from 1.0 --> 1.1 --> 1.2 --> 2.0. Now problem is while upgrading from 1.2 --> 2.0 we saw below errors floodin= g the system.log files in one data center only until we upgraded all the no= des in every DC to 2.0. We think that moment last node was upgraded then th= is error was gone. Does anyone has idea what could have been causing this? = Is it because of some version incompatibility? ERROR [MutationStage:208] 2013-09-21 17:51:08,163 RowMutationVerbHandler.ja= va (line 63) Error in row mutation java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at org.apache.cassandra.net.CompactEndpointSerializationHelper.dese= rialize(CompactEndpointSerializationHelper.java:37) at org.apache.cassandra.db.RowMutationVerbHandler.forwardToLocalNod= es(RowMutationVerbHandler.java:81) at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutatio= nVerbHandler.java:49) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDelivery= Task.java:56) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:615) at java.lang.Thread.run(Thread.java:722) ________________________________ Thanks & Regards Shashilpi Krishan CONFIDENTIALITY NOTICE =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply ema= il and destroy all copies of the original message along with any attachment= s, from your computer system. If you are the intended recipient, please be = advised that the content of this message is subject to access, review and d= isclosure by the sender's Email System Administrator. --_000_4a3240330db04318b06536d0e83d7652BN1PR04MB422namprd04pro_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi Everyone.

 

We had a Cassandra cluster (= running with v1.0.7) spread across 3 data centers with each data center hav= ing 16 nodes. We started upgrading that to 2.0 but realized that we can’t go directly to 2.0 due to read failures, hence to avoi= d down time we have to go from 1.0 =E0 1.1 =E0 1.2 =E0 2.0.

 

Now problem is while upgradi= ng from 1.2 =E0 2.0 we saw below errors flooding the = system.log files in one data center only until we upgraded all the nodes in every DC to 2.0. We think that moment last node was upgra= ded then this error was gone. Does anyone has idea what could have been cau= sing this? Is it because of some version incompatibility?

 

ERROR [MutationStage:208] 2013-09-21 17:51:08,163 RowMuta= tionVerbHandler.java (line 63) Error in row mutation

java.io.EOFException

        at java.io.Dat= aInputStream.readFully(DataInputStream.java:197)

        at org.apache.= cassandra.net.CompactEndpointSerializationHelper.deserialize(CompactEndpoin= tSerializationHelper.java:37)

        at org.apache.= cassandra.db.RowMutationVerbHandler.forwardToLocalNodes(RowMutationVerbHand= ler.java:81)

        at org.apache.= cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:49)<= /span>

        at org.apache.= cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)

        at java.util.c= oncurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)=

        at java.util.c= oncurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)=

        at java.lang.T= hread.run(Thread.java:722)

Thanks &am= p; Regards


Shashilpi Krishan

 


CONFIDENTIALITY NOTICE
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply email and destroy all copies of the ori= ginal message along with any attachments, from your computer system. If you= are the intended recipient, please be advised that the content of this mes= sage is subject to access, review and disclosure by the sender's Email System Administrator.
--_000_4a3240330db04318b06536d0e83d7652BN1PR04MB422namprd04pro_--