cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Sanda <john.sa...@gmail.com>
Subject commit log on NFS volume
Date Tue, 01 Nov 2016 18:10:10 GMT
I know that using NFS is discouraged, particularly for the commit log. Can
anyone shed some light into what kinds of problems I might encounter aside
from performance? The reason for my inquiry is because I have some
deployments with Cassandra 2.2.1 that use NFS and are experiencing some
problems like reoccurring corrupted commit log segments on start up:

    ERROR 19:38:42 Exiting due to error while processing commit log
during initialization.
    org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
            at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
[apache-cassandra-2.2.1.jar]


In one deployment after removing all of corrupted commit log segments I got
a different error:

Exception (java.lang.RuntimeException) encountered during startup:
java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
java.lang.RuntimeException: java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
	at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66)
	at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
	at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
Caused by: java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
	at java.nio.channels.FileChannel.open(FileChannel.java:287)
	at java.nio.channels.FileChannel.open(FileChannel.java:335)
	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
	... 8 more


The latter error looks like it involves compaction and might be unrelated.
I don't know if it matters, but I have commit log compression enabled in
these environments.

-- 

- John

Mime
View raw message