cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Shuler <mich...@pbandjelly.org>
Subject Re: Cassandra Hints file corruption
Date Mon, 09 Apr 2018 19:11:00 GMT
On 04/09/2018 01:43 PM, Vineet G H wrote:
> Hello All,
> 
> We have a 14 node Cassandra cluster 3.11.1. For some odd reason
> intermittently we see the following error
> 
> ERROR [HintsDispatcher:1] 2018-04-06 16:26:44,423
> CassandraDaemon.java:228 - Exception in thread
> Thread[HintsDispatcher:1,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest
> mismatch exception
> at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:298)
> ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]

3.11.1-SNAPSHOT? This could be any commit between the 3.11.0 and 3.11.1
releases (usually). If you built this yourself, what commit sha is your
SNAPSHOT jar from and does the git log show it includes commit f919cf4a4?

Generally, using a release artifact is highly suggested, since everyone
knows the code state of the release. No one but yourself can have any
reasonable knowledge of where your cluster is running at code-wise.

> The jar in question has the patch from bug
> 
> https://issues.apache.org/jira/browse/CASSANDRA-13696
> 
> We are able get past the issue by running truncatehints
> 
> 1. Could these new manifestation of the issue? Probably not related to bug above
> 2. Are there any tools which dump hints file content?
> 3. What are the implication of truncatehints? Sounds like there could
> be data loss, but we have quoram for writes and reads, which means we
> should enough replicas to reconstruct the data.
> 
> I am gathering more evidence on the issue would be happy to work with devs
> 
> Regards,
> Vineet

-- 
Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message