incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: making sense of output from Eclipse Memory Analyzer tool taken from .hprof file
Date Tue, 19 Nov 2013 20:59:44 GMT
What version of cassandra are you using ?
What are the JVM settings? (check with ps aux | grep cassandra)


OOM in cassandra 1.2+ is rare but there is also https://issues.apache.org/jira/browse/CASSANDRA-5706
and https://issues.apache.org/jira/browse/CASSANDRA-6087

> One instance of "org.apache.cassandra.db.ColumnFamilyStore" loaded by "sun.misc.Launcher$AppClassLoader
@ 0x613e1bdc8" occupies 984,094,664 (11.64%) bytes.
938MB is a bit of memory, the CFS and data tracker are dealing with the memtable. This may
indicate things are not being flushed from memory correctly. 

> •java.lang.Thread @ 0x73e1f74c8 CompactionExecutor:158 - 839,225,000 (9.92%) bytes.
> •java.lang.Thread @ 0x717f08178 MutationStage:31 - 809,909,192 (9.58%) bytes.
> •java.lang.Thread @ 0x717f082c8 MutationStage:5 - 649,667,472 (7.68%) bytes.
> •java.lang.Thread @ 0x717f083a8 MutationStage:21 - 498,081,544 (5.89%) bytes.
> •java.lang.Thread @ 0x71b357e70 MutationStage:11 - 444,931,288 (5.26%) bytes.
maybe very big rows and/or very big mutations. 

hope that helps. 

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 15/11/2013, at 12:34 pm, Mike Koh <defmikekoh@gmail.com> wrote:

> I am investigating Java Out of memory heap errors. So I created an .hprof file and loaded
it into Eclipse Memory Analyzer Tool which gave some "Problem Suspects".
> 
> First one looks like:
> ----
> One instance of "org.apache.cassandra.db.ColumnFamilyStore" loaded by "sun.misc.Launcher$AppClassLoader
@ 0x613e1bdc8" occupies 984,094,664 (11.64%) bytes. The memory is accumulated in one instance
of "org.apache.cassandra.db.DataTracker$View" loaded by "sun.misc.Launcher$AppClassLoader
@ 0x613e1bdc8".
> ----
> 
> If I click around into the verbiage, I believe I can pick out the name of a column family
but that is about it. Can someone explain what the above means in more detail and if it is
indicative of a problem?
> 
> 
> Next one looks like:
> -----
> •java.lang.Thread @ 0x73e1f74c8 CompactionExecutor:158 - 839,225,000 (9.92%) bytes.
> •java.lang.Thread @ 0x717f08178 MutationStage:31 - 809,909,192 (9.58%) bytes.
> •java.lang.Thread @ 0x717f082c8 MutationStage:5 - 649,667,472 (7.68%) bytes.
> •java.lang.Thread @ 0x717f083a8 MutationStage:21 - 498,081,544 (5.89%) bytes.
> •java.lang.Thread @ 0x71b357e70 MutationStage:11 - 444,931,288 (5.26%) bytes.
> ------
> If I click into the verbiage, they above Compaction and Mutations all seem to be referencing
the same column family. Are the above related? Is there a way I can tell more exactly what
is being compacted and/or mutated more specifically than which column family?


Mime
View raw message