Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
Date: Thu, 1 Sep 2011 16:54:56 -0700
Message-ID: 
 <CACE81MgpRzr14z4e=w6VEEd-5PDNhHau_OWkMvaF_teENJ9DAQ@mail.gmail.com>
Subject: Fun with Heap Dump ...
From: Ian Danforth <idanforth@numenta.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

All,

 I need help interpreting the results of my investigation. I'm
encountering this error: "Unable to reduce heap usage since there are
no dirty column families". My heap sits near max and occasionally
OOMs. (4GB heap)

Following Mr. Ellis's instructions here:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Memory-Usage-During-Read-td6338635.html

I set the heap down to 1GB, restarted the node, watched the memory
climb in jconsole and waited for a heap dump. (Of course the first
time I tried this I got a permission denied error on writing out the
dump, and had to restart C* as root, but anyway ...)

Below you'll find a screen grab of the heap dump analysis.

http://screencast.com/t/U6IYzloe2McP

Here is what I see in cassandra.log just prior to OOM:

[ec2-user@ip-10-86-223-245 ~]$ tail -f /var/log/cassandra/cassandra.log
 INFO 22:37:11,193 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
 INFO 22:37:11,194 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
 INFO 22:37:11,195 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
 INFO 22:37:11,196 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
 INFO 22:37:11,212 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
 INFO 22:38:32,485 Opening /cassandra/data/Keyspace1/TwitterTest-g-5852
 INFO 22:38:33,253 Opening /cassandra/data/Keyspace1/TwitterTest-g-5502
 INFO 22:38:34,710 Opening /cassandra/data/Keyspace1/TwitterTest-g-5643
 INFO 22:38:35,653 Opening /cassandra/data/Keyspace1/TwitterTest-g-6117
 INFO 22:38:35,699 Opening /cassandra/data/Keyspace1/TwitterTest-g-1376
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor27]
...etc...

>From my org.apache.cassandra.db MBean I get an estimate of 596,504,576 keys.


-----

What I'd really like to know is:

1. What operation is C* performing during lines like these:

 INFO 22:38:34,710 Opening /cassandra/data/Keyspace1/TwitterTest-g-5643

(I think this is an SSTable it's extracting an index for this column
family from)

2. Has my CF index outgrown memory?

3. If so is there a way to relate # CF, # Columns, # Rows to index size?

I need to know how many keys I can store before I need more memory, or
need more nodes.


Thanks in advance. I've been getting a lot of help from the list and I
really appreciate it!

Ian