cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: Attempting to avoid fatal flush due to disk space
Date Wed, 04 Apr 2012 21:31:18 GMT
Hi Aaron,

On Wed, Apr 4, 2012 at 7:57 PM, aaron morton <aaron@thelastpickle.com>wrote:

> Is cleanupDirectoriesFailover able to delete the files ?
>

Yes, around the time that the stack traces appear (after final
teadDownClass()) you'll see my log.INFO statement and subsequent service
log indicating that CassandraDaemon.stop and CassandraDaemon.destroy have
both been called. This is backed up by my manual checks which confirm that
the test directory and files within are sucessfully deleted. So I know that
the cleanupDirectoriesFailover code is executing as planned and that things
are working as they should up until my description as above. I've pasted
the code below with some additional comments below that.

12/04/04 21:59:21 INFO store.DataStoreTestBase: tearing down class
12/04/04 21:59:21 INFO cassandra.GoraCassandraTestDriver: Shutting down
Embedded Cassandra server...
12/04/04 21:59:21 INFO service.AbstractCassandraDaemon: Cassandra shutting
down...
12/04/04 21:59:21 INFO thrift.CassandraDaemon: Stop listening to thrift
clients
Tests run: 28, Failures: 0, Errors: 28, Skipped: 0, Time elapsed: 69.043
sec <<< FAILURE!
12/04/04 22:00:21 INFO net.MessagingService: Shutting down MessageService...
12/04/04 22:00:21 INFO net.MessagingService: MessagingService shutting down
server thread.
12/04/04 22:00:21 INFO net.MessagingService: Waiting for in-progress
requests to complete
12/04/04 22:00:21 INFO db.ColumnFamilyStore: Enqueuing flush of
Memtable-Versions@6196605(83/103 serialized/live bytes, 3 ops)
12/04/04 22:00:21 INFO db.Memtable: Writing Memtable-Versions@6196605(83/103
serialized/live bytes, 3 ops)
12/04/04 22:00:21 ERROR service.AbstractCassandraDaemon: Fatal exception in
thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk
space to flush 133 bytes
    at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133
bytes
    at
org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:623)
    at
org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1850)
    at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:250)
    at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
    at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:291)
    at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    ... 3 more
12/04/04 22:00:21 ERROR service.AbstractCassandraDaemon: Fatal exception in
thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk
space to flush 133 bytes
    at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133
bytes
    at
org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:623)
    at
org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1850)
    at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:250)
    at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
    at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:291)
    at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    ... 3 more


>
> When you get the error is the disk actually full ?
>

I checked the memory on disk it is certainly not full. I have well over
20GB still available!


>
> Can you narrow this down to "i cannot delete the sstables"? (and what
> platform are you on).
>

Well for each failed test I'm currently getting

12/04/04 21:59:21 ERROR thrift.Cassandra$Processor: Internal error
processing system_add_keyspace
java.lang.IncompatibleClassChangeError: Class
org.apache.avro.generic.GenericData$Array does not implement the requested
interface java.util.List
    at org.apache.cassandra.config.KSMetaData.toAvro(KSMetaData.java:136)
    at
org.apache.cassandra.db.migration.Migration.makeDefinitionMutation(Migration.java:223)
    at
org.apache.cassandra.db.migration.AddKeyspace.<init>(AddKeyspace.java:49)
    at
org.apache.cassandra.thrift.CassandraServer.system_add_keyspace(CassandraServer.java:986)
    at
org.apache.cassandra.thrift.Cassandra$Processor$system_add_keyspace.process(Cassandra.java:3912)
    at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
    at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)

This is not merely a failed assertion so I'm currently investigating this
as well.
I'm using single core Ubuntu release 11.10, Kernel Linux 3.0.0-17-generic
GNOME 3.2.1, AMD Athlon 64 processor 3800+

Looking back through my full log they only WARN I have is early on

12/04/04 15:30:09 WARN service.StorageService: Generated random token
122116503163726529681704205068506497955. Random tokens will result in an
unbalanced ring; see http://wiki.apache.org/cassandra/Operations

Although I don't think this leads to the problem I'm encountering. I'm
looking into the 2nd snippet of code I pasted above and hopefully this will
produce some results.

Thanks for your comments.

Lewis

Mime
View raw message