cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vineet G H <vgh4us...@gmail.com>
Subject Re: Cassandra Hints file corruption
Date Mon, 09 Apr 2018 20:15:34 GMT
Yes, the commit log show that we built it out f919cf4a4 which we used
later locally to build it.

I realize that using release artifact is suggested. We tried even
3.11.1 ( official release) and where able to reproduce this issue on
14 node cluster

On Mon, Apr 9, 2018 at 12:11 PM, Michael Shuler <michael@pbandjelly.org> wrote:
> On 04/09/2018 01:43 PM, Vineet G H wrote:
>> Hello All,
>>
>> We have a 14 node Cassandra cluster 3.11.1. For some odd reason
>> intermittently we see the following error
>>
>> ERROR [HintsDispatcher:1] 2018-04-06 16:26:44,423
>> CassandraDaemon.java:228 - Exception in thread
>> Thread[HintsDispatcher:1,1,main]
>> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest
>> mismatch exception
>> at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:298)
>> ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
>
> 3.11.1-SNAPSHOT? This could be any commit between the 3.11.0 and 3.11.1
> releases (usually). If you built this yourself, what commit sha is your
> SNAPSHOT jar from and does the git log show it includes commit f919cf4a4?
>
> Generally, using a release artifact is highly suggested, since everyone
> knows the code state of the release. No one but yourself can have any
> reasonable knowledge of where your cluster is running at code-wise.
>
>> The jar in question has the patch from bug
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-13696
>>
>> We are able get past the issue by running truncatehints
>>
>> 1. Could these new manifestation of the issue? Probably not related to bug above
>> 2. Are there any tools which dump hints file content?
>> 3. What are the implication of truncatehints? Sounds like there could
>> be data loss, but we have quoram for writes and reads, which means we
>> should enough replicas to reconstruct the data.
>>
>> I am gathering more evidence on the issue would be happy to work with devs
>>
>> Regards,
>> Vineet
>
> --
> Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message