cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Danforth <>
Subject Fun with Heap Dump ...
Date Thu, 01 Sep 2011 23:54:56 GMT

 I need help interpreting the results of my investigation. I'm
encountering this error: "Unable to reduce heap usage since there are
no dirty column families". My heap sits near max and occasionally
OOMs. (4GB heap)

Following Mr. Ellis's instructions here:

I set the heap down to 1GB, restarted the node, watched the memory
climb in jconsole and waited for a heap dump. (Of course the first
time I tried this I got a permission denied error on writing out the
dump, and had to restart C* as root, but anyway ...)

Below you'll find a screen grab of the heap dump analysis.

Here is what I see in cassandra.log just prior to OOM:

[ec2-user@ip-10-86-223-245 ~]$ tail -f /var/log/cassandra/cassandra.log
 INFO 22:37:11,193 Removing compacted SSTable files (see
 INFO 22:37:11,194 Removing compacted SSTable files (see
 INFO 22:37:11,195 Removing compacted SSTable files (see
 INFO 22:37:11,196 Removing compacted SSTable files (see
 INFO 22:37:11,212 Removing compacted SSTable files (see
 INFO 22:38:32,485 Opening /cassandra/data/Keyspace1/TwitterTest-g-5852
 INFO 22:38:33,253 Opening /cassandra/data/Keyspace1/TwitterTest-g-5502
 INFO 22:38:34,710 Opening /cassandra/data/Keyspace1/TwitterTest-g-5643
 INFO 22:38:35,653 Opening /cassandra/data/Keyspace1/TwitterTest-g-6117
 INFO 22:38:35,699 Opening /cassandra/data/Keyspace1/TwitterTest-g-1376
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor27]

>From my org.apache.cassandra.db MBean I get an estimate of 596,504,576 keys.


What I'd really like to know is:

1. What operation is C* performing during lines like these:

 INFO 22:38:34,710 Opening /cassandra/data/Keyspace1/TwitterTest-g-5643

(I think this is an SSTable it's extracting an index for this column
family from)

2. Has my CF index outgrown memory?

3. If so is there a way to relate # CF, # Columns, # Rows to index size?

I need to know how many keys I can store before I need more memory, or
need more nodes.

Thanks in advance. I've been getting a lot of help from the list and I
really appreciate it!


View raw message