Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 4BB5C200BC2 for ; Thu, 17 Nov 2016 09:47:39 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 4A13F160B0B; Thu, 17 Nov 2016 08:47:39 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1EA54160AFF for ; Thu, 17 Nov 2016 09:47:37 +0100 (CET) Received: (qmail 96755 invoked by uid 500); 17 Nov 2016 08:47:36 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 96745 invoked by uid 99); 17 Nov 2016 08:47:36 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Nov 2016 08:47:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 2DB2A180538 for ; Thu, 17 Nov 2016 08:47:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -2.049 X-Spam-Level: X-Spam-Status: No, score=-2.049 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 0cADVjrc79ln for ; Thu, 17 Nov 2016 08:47:33 +0000 (UTC) Received: from mailout4-tr.thomsonreuters.com (mailout4-tr.thomsonreuters.com [159.220.20.197]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id BC1F15F47F for ; Thu, 17 Nov 2016 08:47:31 +0000 (UTC) Received: from hk1p-erfsmlr01.erf.thomson.com (relay1 [10.29.195.13]) by mailout4-tr.thomsonreuters.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.1) with ESMTP id uAH8lNZI024008 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 17 Nov 2016 08:47:24 GMT Received: from SG1P-ERFMHUB02.ERF.thomson.com (c443zdm.ten.thomsonreuters.com [10.238.72.19]) by hk1p-erfsmlr01.erf.thomson.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.1) with ESMTP id uAH8lMId004652 for ; Thu, 17 Nov 2016 08:47:23 GMT Received: from SG1P-ERFMMBX03.ERF.thomson.com ([fe80::104c:838c:5edc:d236]) by SG1P-ERFMHUB02.ERF.thomson.com ([fe80::98a:a2d9:38db:81d8%20]) with mapi id 14.03.0235.001; Thu, 17 Nov 2016 08:50:46 +0000 From: To: Subject: lots of DigestMismatchException in cassandra3 Thread-Topic: lots of DigestMismatchException in cassandra3 Thread-Index: AdJArpVMCT3OrM5qSvqbQGV6myD7EA== Date: Thu, 17 Nov 2016 08:47:21 +0000 Message-ID: <2E2FD4ACA211C145911599AE3E6C67860C904C94@SG1P-ERFMMBX03.ERF.thomson.com> Accept-Language: zh-CN, en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.29.134.140] x-tm-as-product-ver: SMEX-11.0.0.4255-8.000.1202-22704.006 x-tm-as-result: No--47.694000-0.000000-31 x-tm-as-user-approved-sender: Yes x-tm-as-user-blocked-sender: No Content-Type: multipart/alternative; boundary="_000_2E2FD4ACA211C145911599AE3E6C67860C904C94SG1PERFMMBX03ER_" MIME-Version: 1.0 X-CFilter-Loop: Reflected archived-at: Thu, 17 Nov 2016 08:47:39 -0000 --_000_2E2FD4ACA211C145911599AE3E6C67860C904C94SG1PERFMMBX03ER_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, We are using Cassandra 3.7 and we have a single datacenter of 4 nodes, RF = =3D3, and the consistency level is quorum. I found lots of DigestMismatchException in debug.log, as below: DEBUG [ReadRepairStage:11] 2016-11-17 08:24:57,066 ReadCallback.java:235 - = Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key Deco= ratedKey(-7187709033646443438, 0008000001586f95380000000405f72c7a0000040000= 000300) (c8d67cfae7be8470ca1786eaa0d8968e vs 3aa974d3eca302682c6f31ca250ea6= 41) at org.apache.cassandra.service.DigestResolver.resolve(Dige= stResolver.java:85) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.service.ReadCallback$AsyncRepairRun= ner.run(ReadCallback.java:226) ~[apache-cassandra-3.7.jar:3.7] at java.util.concurrent.ThreadPoolExecutor.runWorker(Thread= PoolExecutor.java:1142) [na:1.8.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa= dPoolExecutor.java:617) [na:1.8.0_25] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25] INFO [HintsDispatcher:5] 2016-11-17 08:25:06,958 HintsStore.java:126 - Del= eted hint file bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints INFO [HintsDispatcher:5] 2016-11-17 08:25:06,962 HintsDispatchExecutor.jav= a:263 - Finished hinted handoff of file bcd84017-6068-4a76-be89-96c0130599f= 3-1479371095876-1.hints to endpoint bcd84017-6068-4a76-be89-96c0130599f3 DEBUG [GossipStage:1] 2016-11-17 08:25:32,551 FailureDetector.java:456 - Ig= noring interval time of 2003165297 for /10.52.153.107 DEBUG [CompactionExecutor:9] 2016-11-17 08:25:33,034 CompactionTask.java:22= 3 - Compacted (3b3d7600-ac9f-11e6-8b55-03fd515998e4) 4 sstables to [/data/c= 878jrn/data/pc_object_data_beta/vsc_data-53935ff0a74811e69ab46393a19c464a/m= b-2256-big,] to level=3D0. 85.011MiB to 53.580MiB (~63% of original) in 75= ,524ms. Throughput =3D 726.471KiB/s. 0 total partitions merged to 15,552.= Partition merge counts were {1:11623, 2:3543, 3:385, 4:1, } ERROR [MessagingService-Incoming-/10.52.153.109] 2016-11-17 08:25:42,126 Ca= ssandraDaemon.java:217 - Exception in thread Thread[MessagingService-Incomi= ng-/10.52.153.109,5,main] java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered= partition at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerial= izer$1.computeNext(UnfilteredRowIteratorSerializer.java:224) ~[apache-cassa= ndra-3.7.jar:3.7] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerial= izer$1.computeNext(UnfilteredRowIteratorSerializer.java:212) ~[apache-cassa= ndra-3.7.jar:3.7] at org.apache.cassandra.utils.AbstractIterator.hasNext(Abst= ractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.partitions.PartitionUpdate$Parti= tionUpdateSerializer.deserialize30(PartitionUpdate.java:681) ~[apache-cassa= ndra-3.7.jar:3.7] at org.apache.cassandra.db.partitions.PartitionUpdate$Parti= tionUpdateSerializer.deserialize(PartitionUpdate.java:642) ~[apache-cassand= ra-3.7.jar:3.7] at org.apache.cassandra.db.Mutation$MutationSerializer.dese= rialize(Mutation.java:331) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.Mutation$MutationSerializer.dese= rialize(Mutation.java:350) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.Mutation$MutationSerializer.dese= rialize(Mutation.java:287) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:1= 14) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.net.IncomingTcpConnection.receiveMe= ssage(IncomingTcpConnection.java:190) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.net.IncomingTcpConnection.receiveMe= ssages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.net.IncomingTcpConnection.run(Incom= ingTcpConnection.java:92) ~[apache-cassandra-3.7.jar:3.7] Caused by: java.io.IOException: Corrupt empty row found in unfiltered parti= tion at org.apache.cassandra.db.rows.UnfilteredSerializer.deseri= alize(UnfilteredSerializer.java:388) ~[apache-cassandra-3.7.jar:3.7] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerial= izer$1.computeNext(UnfilteredRowIteratorSerializer.java:219) ~[apache-cassa= ndra-3.7.jar:3.7] ... 11 common frames omitted Is this normal? -- Regards, Adeline --_000_2E2FD4ACA211C145911599AE3E6C67860C904C94SG1PERFMMBX03ER_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

 

We are using Cassandra 3.7 and we have a single data= center of 4 nodes, RF =3D3, and the consistency level is quorum.

I found lots of DigestMismatchExceptio= n in debug.log, as below:

DEBUG [ReadRepairStage= :11] 2016-11-17 08:24:57,066 ReadCallback.java:235 - Digest mismatch:<= /o:p>

org.apache.cassandra.s= ervice.DigestMismatchException: Mismatch for key DecoratedKey(-718770903364= 6443438, 0008000001586f95380000000405f72c7a0000040000000300) (c8d67cfae7be8470ca1786eaa0d8968e vs 3aa974d3eca302682c6f31ca250ea641)

   &nbs= p;            at org= .apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) ~[= apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.j= ava:226) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at jav= a.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142= ) [na:1.8.0_25]

   &nbs= p;            at jav= a.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617= ) [na:1.8.0_25]

   &nbs= p;            at jav= a.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

INFO  [HintsDispa= tcher:5] 2016-11-17 08:25:06,958 HintsStore.java:126 - Deleted hint file bc= d84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints<= /p>

INFO  [HintsDispa= tcher:5] 2016-11-17 08:25:06,962 HintsDispatchExecutor.java:263 - Finished = hinted handoff of file bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1= .hints to endpoint bcd84017-6068-4a76-be89-96c0130599f3

DEBUG [GossipStage:1] = 2016-11-17 08:25:32,551 FailureDetector.java:456 - Ignoring interval time o= f 2003165297 for /10.52.153.107

DEBUG [CompactionExecu= tor:9] 2016-11-17 08:25:33,034 CompactionTask.java:223 - Compacted (3b3d760= 0-ac9f-11e6-8b55-03fd515998e4) 4 sstables to [/data/c878jrn/data/pc_object_data_beta/vsc_data-53935ff0a74811e69ab463= 93a19c464a/mb-2256-big,] to level=3D0.  85.011MiB to 53.580MiB (~63% o= f original) in 75,524ms.  Throughput =3D 726.471KiB/s.  0 total p= artitions merged to 15,552.  Partition merge counts were {1:11623, 2:3543, 3:385, 4:1, }

ERROR [MessagingServic= e-Incoming-/10.52.153.109] 2016-11-17 08:25:42,126 CassandraDaemon.java:217= - Exception in thread Thread[MessagingService-Incoming-/10.52.153.109,5,ma= in]

java.io.IOError: java.= io.IOException: Corrupt empty row found in unfiltered partition<= /span>

   &nbs= p;            at org= .apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(Unf= ilteredRowIteratorSerializer.java:224) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(Unf= ilteredRowIteratorSerializer.java:212) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) = ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.d= eserialize30(PartitionUpdate.java:681) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.d= eserialize(PartitionUpdate.java:642) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:= 331) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:= 350) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:= 287) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.net.MessageIn.read(MessageIn.java:114) ~[apache-cassandra= -3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConne= ction.java:190) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConn= ection.java:178) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:= 92) ~[apache-cassandra-3.7.jar:3.7]

Caused by: java.io.IOE= xception: Corrupt empty row found in unfiltered partition=

   &nbs= p;            at org= .apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerial= izer.java:388) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            at org= .apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(Unf= ilteredRowIteratorSerializer.java:219) ~[apache-cassandra-3.7.jar:3.7]

   &nbs= p;            ... 11= common frames omitted

 

Is this normal?

 

 

--

Regards, Adeline

 

 

--_000_2E2FD4ACA211C145911599AE3E6C67860C904C94SG1PERFMMBX03ER_--