cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rohit bhatia <rohit2...@gmail.com>
Subject Re: Cassandra out of Heap memory
Date Mon, 18 Jun 2012 04:18:16 GMT
I am using 1.0.5 . The logs suggest that it was one single instance of
failure and I'm unable to reproduce it.
>From the logs, In a span of 30 seconds, heap usage went from 4.8 gb to
8.8 gb With stop-the-world gc running 20 times. I believe that parNew
was unable to clean up memory due to some problem. I would report if I
am able to reproduce this failure.

On Mon, Jun 18, 2012 at 6:14 AM, aaron morton <aaron@thelastpickle.com> wrote:
> Not commenting on the GC advice but Cassandra memory usage has improved a
> lot since that was written. I would take a look at what was happening and
> see if tweeking Cassandra config helped before modifying GC settings.
>
> "GCInspector.java(line 88): Heap is .9934 full." Is this expected? or
> should I adjust my flush_largest_memtable_at variable.
>
> flush_largetsmemtable_at is a a safety valve only. Reducing it may help avid
> OOM, by it will not treat the cause.
>
> What version are you using ?
>
> 1.0.0 had a an issue where deletes were not taken into consideration
> (https://github.com/apache/cassandra/blob/trunk/CHANGES.txt#L33) but this
> does not sound like the same problem.
>
> Take a look in the logs on the machine and see if it was associated with a
> compaction or repair operation.
>
> I would also consider experimenting on one node with 8GB / 800MB heap sizes.
> More is not always better.
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 14/06/2012, at 8:05 PM, rohit bhatia wrote:
>
> Looking at http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html
> and server logs, I think my situation is this
>
> "The default cassandra settings has the highest peak heap usage. The
> problem with this is that it raises the possibility that during the
> CMS cycle, a collection of the young generation runs out of memory to
> migrate objects to the old generation (a so-called concurrent mode
> failure), leading to stop-the-world full garbage collection. However,
> with a slightly lower setting of the CMS threshold, we get a bit more
> headroom, and more stable overall performance."
>
> I see concurrentMarkSweep system.log Entries trying to gc 2-4 collections.
>
> Any suggestions for preemptive measure for this would be welcome.
>
>

Mime
View raw message