Thanks for following up! On Mon, Oct 19, 2009 at 7:11 PM, Edmond Lau wrote: > I wasn't able to apply the patch and reuse the old tables, but after > nuking the data, I'm no longer running into the issue anymore. > > On Fri, Oct 16, 2009 at 3:16 PM, Jonathan Ellis wrote: >> Those are both consistent with a different bug that was fixed for >> 0.4.1 (now released officially!) >> >> -Jonathan >> >> On Fri, Oct 16, 2009 at 4:59 PM, Edmond Lau wrote: >>> Jonathan - >>> >>> I patched in your latest change that dropped the assertions and tried >>> to restart my cluster on my old data.  2 of 5 nodes still failed to >>> start, with different errors. >>> >>> One dies with a generic EOFException during recovery: >>> >>> INFO - Compacting >>> [/var/lib/cassandra/data/Analytics/movie-53-Data.db,/var/lib/cassandra/data/Analytics/movie-54-Data.db,/var/lib/cassandra/data/Analytics/movie-55-Data.db,/var/lib/cassandra/data/Analytics/movie-56-Data.db] >>> INFO - Completed flushing Memtable(country)@1074231936 >>> ERROR - Exception encountered during startup. >>> java.io.EOFException >>> at java.io.DataInputStream.readFully(DataInputStream.java:180) >>> at java.io.DataInputStream.readUTF(DataInputStream.java:592) >>> at java.io.DataInputStream.readUTF(DataInputStream.java:547) >>> at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:222) >>> at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:320) >>> at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:58) >>> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:91) >>> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:167) >>> Exception encountered during startup. >>> >>> The other dies with a more interesting: >>> >>>  INFO [main] 2009-10-16 21:42:17,082 Memtable.java (line 186) Flushing >>> Memtable(provider)@322343927 >>>  INFO [main] 2009-10-16 21:42:17,179 Memtable.java (line 220) >>> Completed flushing Memtable(provider)@322343927 >>> ERROR [main] 2009-10-16 21:42:17,338 CassandraDaemon.java (line 185) >>> Exception encountered during startup. >>> java.lang.RuntimeException: Unable to load comparator class ''. >>> probably this means you have obsolete sstables lying around >>>        at org.apache.cassandra.db.ColumnFamilySerializer.readComparator(ColumnFamilySerializer.java:108) >>>        at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:83) >>>        at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:229) >>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:320) >>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:58) >>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:91) >>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:167) >>> Caused by: java.lang.ClassNotFoundException: >>>        at java.lang.Class.forName0(Native Method) >>>        at java.lang.Class.forName(Class.java:169) >>>        at org.apache.cassandra.db.ColumnFamilySerializer.readComparator(ColumnFamilySerializer.java:104) >>>        ... 6 more >>> >>> There may be some unprintable garbage characters in the class name. >>> They show up in emacs but aren't visible in the system log file. >>> >>> Would you expect the previous fatal error to have corrupted my data? >>> >>> Edmond >>> >>> On Fri, Oct 16, 2009 at 1:23 PM, Jonathan Ellis wrote: >>>> Don't know if you tried this yet, but Eric noticed a bug in my patch. >>>> There's a new one now, fixed. >>>> >>>> -Jonathan >>>> >>> >> >