Hi Aaron,

On Wed, Apr 4, 2012 at 7:57 PM, aaron morton <aaron@thelastpickle.com> wrote:
Is cleanupDirectoriesFailover able to delete the files ? 

Yes, around the time that the stack traces appear (after final teadDownClass()) you'll see my log.INFO statement and subsequent service log indicating that CassandraDaemon.stop and CassandraDaemon.destroy have both been called. This is backed up by my manual checks which confirm that the test directory and files within are sucessfully deleted. So I know that the cleanupDirectoriesFailover code is executing as planned and that things are working as they should up until my description as above. I've pasted the code below with some additional comments below that.

12/04/04 21:59:21 INFO store.DataStoreTestBase: tearing down class
12/04/04 21:59:21 INFO cassandra.GoraCassandraTestDriver: Shutting down Embedded Cassandra server...
12/04/04 21:59:21 INFO service.AbstractCassandraDaemon: Cassandra shutting down...
12/04/04 21:59:21 INFO thrift.CassandraDaemon: Stop listening to thrift clients
Tests run: 28, Failures: 0, Errors: 28, Skipped: 0, Time elapsed: 69.043 sec <<< FAILURE!
12/04/04 22:00:21 INFO net.MessagingService: Shutting down MessageService...
12/04/04 22:00:21 INFO net.MessagingService: MessagingService shutting down server thread.
12/04/04 22:00:21 INFO net.MessagingService: Waiting for in-progress requests to complete
12/04/04 22:00:21 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Versions@6196605(83/103 serialized/live bytes, 3 ops)
12/04/04 22:00:21 INFO db.Memtable: Writing Memtable-Versions@6196605(83/103 serialized/live bytes, 3 ops)
12/04/04 22:00:21 ERROR service.AbstractCassandraDaemon: Fatal exception in thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
    at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:623)
    at org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1850)
    at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:250)
    at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
    at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:291)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    ... 3 more
12/04/04 22:00:21 ERROR service.AbstractCassandraDaemon: Fatal exception in thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
    at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:623)
    at org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1850)
    at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:250)
    at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
    at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:291)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    ... 3 more
 

When you get the error is the disk actually full ?

I checked the memory on disk it is certainly not full. I have well over 20GB still available!
 

Can you narrow this down to "i cannot delete the sstables"? (and what platform are you on). 

Well for each failed test I'm currently getting

12/04/04 21:59:21 ERROR thrift.Cassandra$Processor: Internal error processing system_add_keyspace
java.lang.IncompatibleClassChangeError: Class org.apache.avro.generic.GenericData$Array does not implement the requested interface java.util.List
    at org.apache.cassandra.config.KSMetaData.toAvro(KSMetaData.java:136)
    at org.apache.cassandra.db.migration.Migration.makeDefinitionMutation(Migration.java:223)
    at org.apache.cassandra.db.migration.AddKeyspace.<init>(AddKeyspace.java:49)
    at org.apache.cassandra.thrift.CassandraServer.system_add_keyspace(CassandraServer.java:986)
    at org.apache.cassandra.thrift.Cassandra$Processor$system_add_keyspace.process(Cassandra.java:3912)
    at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)

This is not merely a failed assertion so I'm currently investigating this as well.
I'm using single core Ubuntu release 11.10, Kernel Linux 3.0.0-17-generic GNOME 3.2.1, AMD Athlon 64 processor 3800+

Looking back through my full log they only WARN I have is early on

12/04/04 15:30:09 WARN service.StorageService: Generated random token 122116503163726529681704205068506497955. Random tokens will result in an unbalanced ring; see http://wiki.apache.org/cassandra/Operations

Although I don't think this leads to the problem I'm encountering. I'm looking into the 2nd snippet of code I pasted above and hopefully this will produce some results.

Thanks for your comments.

Lewis