cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tommy Stendahl <tommy.stend...@ericsson.com>
Subject Re: lots of DigestMismatchException in cassandra3
Date Thu, 17 Nov 2016 09:16:10 GMT
Hi,

Could it be CASSANDRA-12090 
<https://issues.apache.org/jira/browse/CASSANDRA-12090>?

Regards,
/Tommy

On 2016-11-17 09:47, Adeline.Pan@thomsonreuters.com wrote:
>
> Hi,
>
> We are using Cassandra 3.7 and we have a single datacenter of 4 nodes, 
> RF =3, and the consistency level is quorum.
>
> I found *lots* *of* DigestMismatchException in debug.log, as below:
>
> DEBUG [ReadRepairStage:11] 2016-11-17 08:24:57,066 
> ReadCallback.java:235 - Digest mismatch:
>
> org.apache.cassandra.service.DigestMismatchException: Mismatch for key 
> DecoratedKey(-7187709033646443438, 
> 0008000001586f95380000000405f72c7a0000040000000300) 
> (c8d67cfae7be8470ca1786eaa0d8968e vs 3aa974d3eca302682c6f31ca250ea641)
>
> at 
> org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) 
> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226)

> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
> [na:1.8.0_25]
>
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
> [na:1.8.0_25]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]
>
> INFO [HintsDispatcher:5] 2016-11-17 08:25:06,958 HintsStore.java:126 - 
> Deleted hint file 
> bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints
>
> INFO [HintsDispatcher:5] 2016-11-17 08:25:06,962 
> HintsDispatchExecutor.java:263 - Finished hinted handoff of file 
> bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints to endpoint 
> bcd84017-6068-4a76-be89-96c0130599f3
>
> DEBUG [GossipStage:1] 2016-11-17 08:25:32,551 FailureDetector.java:456 
> - Ignoring interval time of 2003165297 for /10.52.153.107
>
> DEBUG [CompactionExecutor:9] 2016-11-17 08:25:33,034 
> CompactionTask.java:223 - Compacted 
> (3b3d7600-ac9f-11e6-8b55-03fd515998e4) 4 sstables to 
> [/data/c878jrn/data/pc_object_data_beta/vsc_data-53935ff0a74811e69ab46393a19c464a/mb-2256-big,]

> to level=0.  85.011MiB to 53.580MiB (~63% of original) in 75,524ms.  
> Throughput = 726.471KiB/s. 0 total partitions merged to 15,552.  
> Partition merge counts were {1:11623, 2:3543, 3:385, 4:1, }
>
> ERROR [MessagingService-Incoming-/10.52.153.109] 2016-11-17 
> 08:25:42,126 CassandraDaemon.java:217 - Exception in thread 
> Thread[MessagingService-Incoming-/10.52.153.109,5,main]
>
> java.io.IOError: java.io.IOException: Corrupt empty row found in 
> unfiltered partition
>
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)

> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)

> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)

> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)

> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:331) 
> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:350) 
> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:287) 
> ~[apache-cassandra-3.7.jar:3.7]
>
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) 
> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)

> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)

> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) 
> ~[apache-cassandra-3.7.jar:3.7]
>
> Caused by: java.io.IOException: Corrupt empty row found in unfiltered 
> partition
>
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:388)

> ~[apache-cassandra-3.7.jar:3.7]
>
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219)

> ~[apache-cassandra-3.7.jar:3.7]
>
> ... 11 common frames omitted
>
> Is this normal?
>
> --
>
> Regards, Adeline
>


Mime
View raw message