cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ivan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-2930) corrupt commitlog
Date Thu, 21 Jul 2011 12:35:58 GMT
corrupt commitlog
-----------------

                 Key: CASSANDRA-2930
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.8.1
         Environment: Linux, amd64.
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
            Reporter: ivan


We get "Exception encountered during startup" error while Cassandra starts.

Error messages:
 INFO 13:56:28,736 Finished reading /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
ERROR 13:56:28,736 Exception encountered during startup.
java.io.IOError: java.io.EOFException
        at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
        at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
        at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
        at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
        at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
        at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
        at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
        at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
        at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
        at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
        at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
        at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
        ... 13 more
Exception encountered during startup.
java.io.IOError: java.io.EOFException
        at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
        at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
        at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
        at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
        at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
        at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
        at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
        at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
        at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
        at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
        at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
        at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
        ... 13 more

After some debugging I found that in some serialized supercolumns column counter is less than
the number of serialized columns. Difference was always 1 in corrupt commitlogs. This error
always appears with supercolumns with more than one column, but there are properly serialized
supercolumns also in commitlog.

I have no clue yet why this error happens. I suspect it maybe a race condition.







--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message