cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Caleb Rackliffe <ca...@steelhouse.com>
Subject Re: OutOfMemory Errors with Cassandra 1.0.5 (fixed)
Date Sat, 07 Jan 2012 01:59:48 GMT
Okay, it looks like I was slightly underestimating the number of connections open on the cluster.
 This probably won't be a problem after I tighten up the Hector pool maximums.

Sorry for the spam…

Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com


From: Caleb Rackliffe <caleb@steelhouse.com<mailto:caleb@steelhouse.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Fri, 6 Jan 2012 20:13:37 -0500
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: OutOfMemory Errors with Cassandra 1.0.5

I saw this article - http://comments.gmane.org/gmane.comp.db.cassandra.user/2225

I'm using the Hector client (for connection pooling), with ~3200 threads active according
to JConsole.

Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<mailto:caleb@steelhouse.com>

From: Caleb Rackliffe <caleb@steelhouse.com<mailto:caleb@steelhouse.com>>
Date: Fri, 6 Jan 2012 15:40:26 -0500
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: OutOfMemory Errors with Cassandra 1.0.5

One other item…

java –version

java version "1.7.0_01"
Java(TM) SE Runtime Environment (build 1.7.0_01-b08)
Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)

Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<mailto:caleb@steelhouse.com>


From: Caleb Rackliffe <caleb@steelhouse.com<mailto:caleb@steelhouse.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Fri, 6 Jan 2012 15:28:30 -0500
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: OutOfMemory Errors with Cassandra 1.0.5

Hi Everybody,

I have a 10-node cluster running 1.0.5.  The hardware/configuration for each box looks like
this:

Hardware: 4 GB RAM, 400 GB SATAII HD for commitlog, 50 GB SATAIII SSD for data directory,
1 GB SSD swap partition
OS: CentOS 6, vm.swapiness = 0
Cassandra: disk access mode = standard, max memtable size = 128 MB, max new heap = 800 MB,
max heap = 2 GB, stack size = 128k

I explicitly didn't put JNA on the classpath because I had a hard time figuring out how much
native memory it would actually need.

After a node runs for a couple of days, my swap partition is almost completely full, and even
though the resident size of my Java process is right under 3 GB, I get this sequence in the
logs, with death coming on a failure to allocate another thread…

 WARN [pool-1-thread-1] 2012-01-05 09:06:38,078 Memtable.java (line 174) setting live ratio
to maximum of 64 instead of 65.58206914005034
 WARN [pool-1-thread-1] 2012-01-05 09:08:14,405 Memtable.java (line 174) setting live ratio
to maximum of 64 instead of 1379.0945945945946
 WARN [ScheduledTasks:1] 2012-01-05 09:08:31,593 GCInspector.java (line 146) Heap is 0.7523060581548427
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 WARN [ScheduledTasks:1] 2012-01-05 09:08:31,611 StorageService.java (line 2535) Flushing
CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
 WARN [pool-1-thread-1] 2012-01-05 13:45:29,934 Memtable.java (line 169) setting live ratio
to minimum of 1.0 instead of 0.004297106677189052
 WARN [pool-1-thread-1] 2012-01-06 02:23:18,175 Memtable.java (line 169) setting live ratio
to minimum of 1.0 instead of 0.0018187309961539236
 WARN [ScheduledTasks:1] 2012-01-06 06:10:05,202 GCInspector.java (line 146) Heap is 0.7635993298476305
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 WARN [ScheduledTasks:1] 2012-01-06 06:10:05,203 StorageService.java (line 2535) Flushing
CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
 WARN [ScheduledTasks:1] 2012-01-06 14:59:49,588 GCInspector.java (line 146) Heap is 0.7617639564886326
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 WARN [ScheduledTasks:1] 2012-01-06 14:59:49,612 StorageService.java (line 2535) Flushing
CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
ERROR [CompactionExecutor:6880] 2012-01-06 19:45:49,336 AbstractCassandraDaemon.java (line
133) Fatal exception in thread Thread[CompactionExecutor:6880,1,main]
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:691)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1325)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
        at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getCompactedRow(ParallelCompactionIterable.java:190)
        at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:164)
        at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:144)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:116)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:103)
        at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:90)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172)
        at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

Has anybody seen this sort of problem before?

Thanks to anyone who takes a look.  I can provide more information than this, but I figure
that's enough to start…

Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<mailto:caleb@steelhouse.com>

Mime
View raw message