cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Adeline....@thomsonreuters.com>
Subject lots of DigestMismatchException in cassandra3
Date Thu, 17 Nov 2016 08:47:21 GMT
Hi,

We are using Cassandra 3.7 and we have a single datacenter of 4 nodes, RF =3, and the consistency
level is quorum.
I found lots of DigestMismatchException in debug.log, as below:
DEBUG [ReadRepairStage:11] 2016-11-17 08:24:57,066 ReadCallback.java:235 - Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(-7187709033646443438,
0008000001586f95380000000405f72c7a0000040000000300) (c8d67cfae7be8470ca1786eaa0d8968e vs 3aa974d3eca302682c6f31ca250ea641)
                at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226)
~[apache-cassandra-3.7.jar:3.7]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_25]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_25]
                at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]
INFO  [HintsDispatcher:5] 2016-11-17 08:25:06,958 HintsStore.java:126 - Deleted hint file
bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints
INFO  [HintsDispatcher:5] 2016-11-17 08:25:06,962 HintsDispatchExecutor.java:263 - Finished
hinted handoff of file bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints to endpoint
bcd84017-6068-4a76-be89-96c0130599f3
DEBUG [GossipStage:1] 2016-11-17 08:25:32,551 FailureDetector.java:456 - Ignoring interval
time of 2003165297 for /10.52.153.107
DEBUG [CompactionExecutor:9] 2016-11-17 08:25:33,034 CompactionTask.java:223 - Compacted (3b3d7600-ac9f-11e6-8b55-03fd515998e4)
4 sstables to [/data/c878jrn/data/pc_object_data_beta/vsc_data-53935ff0a74811e69ab46393a19c464a/mb-2256-big,]
to level=0.  85.011MiB to 53.580MiB (~63% of original) in 75,524ms.  Throughput = 726.471KiB/s.
 0 total partitions merged to 15,552.  Partition merge counts were {1:11623, 2:3543, 3:385,
4:1, }
ERROR [MessagingService-Incoming-/10.52.153.109] 2016-11-17 08:25:42,126 CassandraDaemon.java:217
- Exception in thread Thread[MessagingService-Incoming-/10.52.153.109,5,main]
java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered partition
                at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:331)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:350)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:287)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) ~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.7.jar:3.7]
Caused by: java.io.IOException: Corrupt empty row found in unfiltered partition
                at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:388)
~[apache-cassandra-3.7.jar:3.7]
                at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219)
~[apache-cassandra-3.7.jar:3.7]
                ... 11 common frames omitted


Is this normal?


--
Regards, Adeline



Mime
View raw message