cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sébastien Giroux (JIRA) <j...@apache.org>
Subject [jira] Issue Comment Edited: (CASSANDRA-2273) Possible Memory leak
Date Fri, 04 Mar 2011 19:03:37 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002781#comment-13002781
] 

Sébastien Giroux edited comment on CASSANDRA-2273 at 3/4/11 7:02 PM:
---------------------------------------------------------------------

Attached one more screenshot of jconsole. OOM then restart and start using all the heap right
away.

Here is the OOM stacktrace. This time it seems to be while compacting ?

ERROR 18:49:04,580 Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:121)
        at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
        at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2145)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:448)
        at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:240)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
ERROR 18:49:04,580 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:356)
        at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
        at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:409)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap
space
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:352)
        ... 6 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:121)
        at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
        at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2145)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:448)
        at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:240)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        ... 3 more
ERROR 18:49:04,989 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:356)
        at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
        at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:409)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap
space
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:352)
        ... 6 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:121)
        at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
        at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2145)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:448)
        at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:240)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        ... 3 more
Service exit with a return value of 100


      was (Author: wajam):
    Attached one more screenshot of jconsole. OOM then restart and start using all the heap
right away.

Here is the OOM stacktrace

ERROR 18:49:04,580 Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:121)
        at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
        at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2145)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:448)
        at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:240)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
ERROR 18:49:04,580 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:356)
        at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
        at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:409)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap
space
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:352)
        ... 6 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:121)
        at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
        at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2145)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:448)
        at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:240)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        ... 3 more
ERROR 18:49:04,989 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:356)
        at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
        at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:409)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap
space
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:352)
        ... 6 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:121)
        at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
        at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2145)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:448)
        at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:240)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        ... 3 more
Service exit with a return value of 100

  
> Possible Memory leak
> --------------------
>
>                 Key: CASSANDRA-2273
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2273
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Sébastien Giroux
>            Priority: Critical
>             Fix For: 0.7.4
>
>         Attachments: heap_peak_OOM.PNG, jconsole_OOM.PNG
>
>
> I have a few problematic nodes in my cluster that will crash OutOfMemory very often.
This is Cassandra 0.7.3 downloaded from Hudson.
> Heap size is 6GB, server memory is 8GB.
> Memtable are flushed at 64MB, I have 5 CFs.
> FlushLargestMemtablesAt is set at 0.8 but doesn't help with this issue.
> I will attach a screenshot showing my issue. There is no compaction going on when the
heap usage start increasing like crazy.
> It could be a configuration issue but it kinda looks like a bug to me.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message