Hi,

We are using cassandra 0.6.5 to store some analysis data. Recently we added a timer task to remove data older than a month and deployed on a test setup. 
The removal worked fine but immediately after the data was removed cassandra node got OutOfMemory in compaction thread. Heap size is configured as 1G on the test machine and 8G on prod machine.
  1. Is slowing down the deletes from the timer task necessary (and recommended practice) to avoid this (even after increasing heap size)? 
  2. Even after the uncaught exception GC Inspection thread kept running. (any possible reasons for this)

ERROR [COMPACTION-POOL:1] 2010-09-30 01:05:26,457 CassandraDaemon.java (line 86) Uncaught exception in thread Thread[COMPACTION-POOL:1,5,main]
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
        at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:577)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.fillBuffer(BufferedRandomAccessFile.java:210)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.seek(BufferedRandomAccessFile.java:247)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.writeAtMost(BufferedRandomAccessFile.java:390)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.write(BufferedRandomAccessFile.java:366)
        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
        at java.io.RandomAccessFile.writeUTF(RandomAccessFile.java:1064)
        at org.apache.cassandra.io.SSTableWriter.append(SSTableWriter.java:96)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:300)
        at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
        at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        ... 2 more
 INFO [GC inspection] 2010-09-30 01:05:26,462 GCInspector.java (line 156) FLUSH-WRITER-POOL                 0         0
 INFO [GC inspection] 2010-09-30 01:05:26,523 GCInspector.java (line 161) CompactionManager               n/a         1
 INFO [GC inspection] 2010-09-30 03:48:54,613 GCInspector.java (line 129) GC for ParNew: 609 ms, 34948216 reclaimed leaving 819991944 used; max is 1207828480
 INFO [GC inspection] 2010-09-30 04:49:46,435 GCInspector.java (line 129) GC for ParNew: 4561 ms, 34929320 reclaimed leaving 819993312 used; max is 1207828480


--
Regards,

Sudhir Jorwekar