incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sudhir j <sudhir.jorwe...@gmail.com>
Subject OutOfMemory on deletes
Date Fri, 01 Oct 2010 06:36:09 GMT
Hi,

We are using cassandra 0.6.5 to store some analysis data. Recently we added
a timer task to remove data older than a month and deployed on a test
setup.
The removal worked fine but immediately after the data was removed cassandra
node got OutOfMemory in compaction thread. Heap size is configured as 1G on
the test machine and 8G on prod machine.

   1. Is slowing down the deletes from the timer task necessary (and
   recommended practice) to avoid this (even after increasing heap size)?
   2. Even after the uncaught exception GC Inspection thread kept running.
   (any possible reasons for this)


ERROR [COMPACTION-POOL:1] 2010-09-30 01:05:26,457 CassandraDaemon.java (line
86) Uncaught exception in thread Thread[COMPACTION-POOL:1,5,main]
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
        at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
        at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:577)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.fillBuffer(BufferedRandomAccessFile.java:210)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.seek(BufferedRandomAccessFile.java:247)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.writeAtMost(BufferedRandomAccessFile.java:390)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.write(BufferedRandomAccessFile.java:366)
        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
        at java.io.RandomAccessFile.writeUTF(RandomAccessFile.java:1064)
        at
org.apache.cassandra.io.SSTableWriter.append(SSTableWriter.java:96)
        at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:300)
        at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
        at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        ... 2 more
 INFO [GC inspection] 2010-09-30 01:05:26,462 GCInspector.java (line 156)
FLUSH-WRITER-POOL                 0         0
 INFO [GC inspection] 2010-09-30 01:05:26,523 GCInspector.java (line 161)
CompactionManager               n/a         1
 INFO [GC inspection] 2010-09-30 03:48:54,613 GCInspector.java (line 129) GC
for ParNew: 609 ms, 34948216 reclaimed leaving 819991944 used; max is
1207828480
 INFO [GC inspection] 2010-09-30 04:49:46,435 GCInspector.java (line 129) GC
for ParNew: 4561 ms, 34929320 reclaimed leaving 819993312 used; max is
1207828480


-- 
Regards,

Sudhir Jorwekar

Mime
View raw message