cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Gupta <var...@uber.com>
Subject Re: Corrupted commit log prevents Cassandra start
Date Sun, 09 Jul 2017 04:01:36 GMT
If you already have a regular cadence of repair then you can set
"commit_failure_policy" to ignore in cassandra.yaml. So that C* process
does not crash on corrupt commit log.

On Fri, Jul 7, 2017 at 2:10 AM, Hannu Kröger <hkroger@gmail.com> wrote:

> Hello,
>
> yes, that’s what we do when things like this happen.
>
> My thinking is just that when commit log is corrupted, you cannot really
> do anything else but exactly those steps. Delete corrupted file and run
> repair after starting. At least I haven’t heard of any tools for salvaging
> commit log sections.
>
> Current behaviour gives DBA control over when to do those things and of
> course DBA realizes this way that things didn’t go ok but that’s about it.
> There is no alternative way of healing the system or anything.
>
> Hannu
>
> On 7 July 2017 at 12:03:06, benjamin roth (brstgt@gmail.com) wrote:
>
> Hi Hannu,
>
> I remember there have been discussions about this in the past. Most
> probably there is already a JIRA for this.
> I roughly remember a consense like that:
> - Default behaviour should remain
> - It should be configurable to the needs and preferences of the DBA
> - It should at least spit out errors in the logs
>
> ... of course it would be even better to have the underlying issue fixed
> that commit logs should not be corrupt but I remember that this is not so
> easy due to some "architectural implications" of Cassandra. IIRC Ed
> Capriolo posted something related to that some months ago.
>
> For a quick fix, I'd recommend:
> - Delete the affected log file
> - Start the node
> - Run a full-range (not -pr) repair on that node
>
> 2017-07-07 10:57 GMT+02:00 Hannu Kröger <hkroger@gmail.com>:
>
>> Hello,
>>
>> We had a test server crashing for some reason (not related to Cassandra
>> probably) and now when trying to start cassandra, it gives following error:
>>
>> ERROR [main] 2017-07-06 09:29:56,140 JVMStabilityInspector.java:82 -
>> Exiting due to error while processing commit log during initialization.
>> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
>> Mutation checksum failure at 24240116 in Next section at 24239690 in
>> CommitLog-6-1498576271195.log
>> at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:332)
>> [apache-cassandra-3.10.jar:3.10]
>> at org.apache.cassandra.db.commitlog.CommitLogReader.readCommit
>> LogSegment(CommitLogReader.java:201) [apache-cassandra-3.10.jar:3.10]
>> at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:84)
>> [apache-cassandra-3.10.jar:3.10]
>> at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFi
>> les(CommitLogReplayer.java:140) [apache-cassandra-3.10.jar:3.10]
>> at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:177)
>> [apache-cassandra-3.10.jar:3.10]
>> at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:158)
>> [apache-cassandra-3.10.jar:3.10]
>> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:326)
>> [apache-cassandra-3.10.jar:3.10]
>> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601)
>> [apache-cassandra-3.10.jar:3.10]
>> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735)
>> [apache-cassandra-3.10.jar:3.10]
>>
>> Shouldn’t Cassandra tolerate this situation?
>>
>> Of course we can delete commit logs and life goes on. But isn’t this a
>> bug or something?
>>
>> Hannu
>>
>>
>

Mime
View raw message