cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: OutOfMemory on deletes
Date Fri, 01 Oct 2010 13:39:51 GMT
this looks like compaction is OOMing while processing a large row.

On Fri, Oct 1, 2010 at 1:36 AM, Sudhir j <sudhir.jorwekar@gmail.com> wrote:
> Hi,
> We are using cassandra 0.6.5 to store some analysis data. Recently we added
> a timer task to remove data older than a month and deployed on a test
> setup.
> The removal worked fine but immediately after the data was removed cassandra
> node got OutOfMemory in compaction thread. Heap size is configured as 1G on
> the test machine and 8G on prod machine.
>
> Is slowing down the deletes from the timer task necessary (and recommended
> practice) to avoid this (even after increasing heap size)?
> Even after the uncaught exception GC Inspection thread kept running. (any
> possible reasons for this)
>
> ERROR [COMPACTION-POOL:1] 2010-09-30 01:05:26,457 CassandraDaemon.java (line
> 86) Uncaught exception in thread Thread[COMPACTION-POOL:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>         at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>         at
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:577)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.OutOfMemoryError
>         at java.io.RandomAccessFile.readBytes(Native Method)
>         at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
>         at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.fillBuffer(BufferedRandomAccessFile.java:210)
>         at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.seek(BufferedRandomAccessFile.java:247)
>         at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.writeAtMost(BufferedRandomAccessFile.java:390)
>         at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.write(BufferedRandomAccessFile.java:366)
>         at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
>         at java.io.RandomAccessFile.writeUTF(RandomAccessFile.java:1064)
>         at
> org.apache.cassandra.io.SSTableWriter.append(SSTableWriter.java:96)
>         at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:300)
>         at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
>         at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         ... 2 more
>  INFO [GC inspection] 2010-09-30 01:05:26,462 GCInspector.java (line 156)
> FLUSH-WRITER-POOL                 0         0
>  INFO [GC inspection] 2010-09-30 01:05:26,523 GCInspector.java (line 161)
> CompactionManager               n/a         1
>  INFO [GC inspection] 2010-09-30 03:48:54,613 GCInspector.java (line 129) GC
> for ParNew: 609 ms, 34948216 reclaimed leaving 819991944 used; max is
> 1207828480
>  INFO [GC inspection] 2010-09-30 04:49:46,435 GCInspector.java (line 129) GC
> for ParNew: 4561 ms, 34929320 reclaimed leaving 819993312 used; max is
> 1207828480
>
> --
> Regards,
>
> Sudhir Jorwekar
>
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message