cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-2604) EOFException on commitlogs
Date Thu, 05 May 2011 13:49:03 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-2604:
----------------------------------------

    Attachment: 0001-avoid-modifying-original-mutation-during-apply.patch

Terje, do you use secondary indexes in your tests ?

I think I've found a race (related to secondary indexes) that would explain the observed behavior.
But without a good way to reproduce, hard to say if that is the problem.

The problem is in Table.apply(), where ignoreObsoleteMutations() may remove some columns from
the original mutation. And since the commit log aliases the same mutation and, unless you
are in batch mode, can write it anytime during apply(), it is possible for a column to be
removed between the time where the commit log serialize the number of columns and the actual
serialization of the columns (explaining why there is less columns than advertised on replay).

Attaching a patch that lazily clone the column family in ignoreObsoleteMutation to avoid modifying
the original one. Note that 0.7 is not concerned, because in 0.7 we give to the commit log
the mutation already serialized.

> EOFException on commitlogs
> --------------------------
>
>                 Key: CASSANDRA-2604
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2604
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8 beta 1
>            Reporter: Terje Marthinussen
>            Assignee: Sylvain Lebresne
>             Fix For: 0.8.0
>
>         Attachments: 0001-avoid-modifying-original-mutation-during-apply.patch
>
>
> I have seen this occasionally since we started testing 0.8.
> It happens when reading commitlogs on startups.
> However, I have seen it a lot less on 0.8 beta2 (although this is from beta 2)
> ERROR [main] 2011-05-04 18:02:38,134 AbstractCassandraDaemon.java (line 330) Exception
encountered during startup.
> java.io.EOFException
> 	at java.io.DataInputStream.readByte(DataInputStream.java:250)
> 	at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:357)
> 	at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> 	at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:252)
> 	at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:43)
> 	at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136)
> 	at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:368)
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:256)
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:157)
> 	at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173)
> 	at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313)
> 	at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Note that the line numbers on columnserializer may be off due to some local changes,
but those changes are in code not executed in this case and I am 100% sure they do not trigger
this problem.
> I looked on this in the debugger in eclipse on a trunk from 0.8 2 weeks ago, and the
interesting thing I saw was that according to the debugger, the offset of the inputstream
to the deserializer was already at the end (very last byte) of the underlying bytebuffer but
according to the stack, it was trying to read the length of the column name (first read done
in the deserialized).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message