Is cleanupDirectoriesFailover able to delete the files ? 

When you get the error is the disk actually full ?

Can you narrow this down to "i cannot delete the sstables"? (and what platform are you on). 

Cheers
 
-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 5/04/2012, at 1:57 AM, Lewis John Mcgibbney wrote:

Hi,

When writing some test code (to test our Gora-Cassandra module) for Apache Gora, I'm experiencing a major problem when trying to tearDown, flush and basically clean everything up. The tests consist of running our test suite against an embedded Cassandra instance using the Gora API to do all sorts of tasks which may highkight where there are bugs in the code.

Now regardless of whether the tests are failing or not, I get the log and subsequent stack trace as below which quite clearly indicates that there is insufficient disk to flush. However as I don't have the luxury of making changes @ runtime I'm slightly stumped as to how to solve it as all I'm working with is the Cassandra server which is initiated in setUpClass.

My yaml is ripped directly from the Cassandra 1.0.2 distribution so there is nothing out of place in there. However maybe there are some settings which I have not configured?

Please indicate if I need to provide some more information to paint a clearer picture of the situation.

When calling tearDownClass I have

        if (cassandraThread != null) {
            cassandraDaemon.stop();
            cassandraDaemon.destroy();
            cassandraThread.interrupt();
            cassandraThread = null;
        }
        cleanupDirectoriesFailover(); // this recursively deletes the directories Cassandra works with whilst the server is running.
    }

Thank you very much in advance for any help coming this way.

Lewis

12/04/04 13:24:11 INFO gora.GoraTestDriver: tearing down test
12/04/04 13:24:11 INFO migration.Migration: Applying migration 15541bf0-7e51-11e1-0000-242d50cf1fff Drop keyspace: Employee
12/04/04 13:24:11 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Migrations@9190262(6586/82325 serialized/live bytes, 1 ops)
12/04/04 13:24:11 INFO db.Memtable: Writing Memtable-Migrations@9190262(6586/82325 serialized/live bytes, 1 ops)
12/04/04 13:24:11 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Schema@9695615(2796/34950 serialized/live bytes, 2 ops)
12/04/04 13:24:11 INFO db.Memtable: Completed flushing target/test/var/lib/cassandra/data/system/Migrations-h-2764-Data.db (6650 bytes)
12/04/04 13:24:11 INFO db.Memtable: Writing Memtable-Schema@9695615(2796/34950 serialized/live bytes, 2 ops)
12/04/04 13:24:11 INFO db.Memtable: Completed flushing target/test/var/lib/cassandra/data/system/Schema-h-2764-Data.db (2946 bytes)
12/04/04 13:24:12 INFO compaction.CompactionTask: Compacted to [target/test/var/lib/cassandra/data/system/Migrations-h-2765-Data.db,].  12,979,714 to 12,979,522 (~99% of original) bytes for 1 keys at 12.341213MB/s.  Time: 1,003ms.
12/04/04 13:24:12 INFO compaction.CompactionTask: Compacting [SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2764-Data.db'), SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2760-Data.db'), SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2761-Data.db'), SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2763-Data.db')]
12/04/04 13:24:12 INFO store.DataStoreTestBase: tearing down class
12/04/04 13:24:12 INFO service.AbstractCassandraDaemon: Cassandra shutting down...
12/04/04 13:24:12 INFO thrift.CassandraDaemon: Stop listening to thrift clients
12/04/04 13:24:12 INFO compaction.CompactionTask: Compacted to [target/test/var/lib/cassandra/data/system/Schema-h-2765-Data.db,].  5,735,956 to 5,735,629 (~99% of original) bytes for 1,681 keys at 9.011404MB/s.  Time: 607ms.
Tests run: 51, Failures: 1, Errors: 50, Skipped: 0, Time elapsed: 913.842 sec <<< FAILURE!
12/04/04 13:25:12 INFO net.MessagingService: Shutting down MessageService...
12/04/04 13:25:12 INFO net.MessagingService: MessagingService shutting down server thread.
12/04/04 13:25:12 INFO net.MessagingService: Waiting for in-progress requests to complete
12/04/04 13:25:12 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Versions@6081747(83/103 serialized/live bytes, 3 ops)
12/04/04 13:25:12 INFO db.Memtable: Writing Memtable-Versions@6081747(83/103 serialized/live bytes, 3 ops)
12/04/04 13:25:12 ERROR service.AbstractCassandraDaemon: Fatal exception in thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
    at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:621)
    at org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1866)
    at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:248)
    at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
    at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    ... 3 more
12/04/04 13:25:12 ERROR service.AbstractCassandraDaemon: Fatal exception in thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
    at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:621)
    at org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1866)
    at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:248)
    at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
    at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    ... 3 more

--
Lewis