incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arindam Barua <aba...@247-inc.com>
Subject Heap almost full
Date Wed, 16 Oct 2013 00:21:14 GMT

During performance testing being run on our 4 node Cassandra 1.1.5 cluster, we are seeing
warning logs about the heap being almost full - [1]. I'm trying to figure out why, and how
to prevent it.

The tests are being run on a Cassandra ring consisting of 4 dedicated boxes with 32 GB of
RAM each.
The heap size is set to 8 GB as recommended.
All the other relevant settings I know off are the default ones:

-          memtable_total_space_in_mb is not set in the yaml, so should default to 1/3rd the
heap size.

-          They key cache should be 100 MB at the most. I checked the key cache the day after
the tests were run via nodetool info, and it reported 4.5 MB being used.

-          row cache is not being used

-          I summed up the bloom filter usage reported by nodetool cfstats in all the CFs
and it was under 50 MB.

The resident size of the cassandra process accd to top is 8.4g even now. Did a heap histogram
using jmap, but not sure how to interpret those results usefully - [2].

Performance test details:

-          The test is write only, and is writing relatively large amount of data to one CF.

-          There is some other traffic that is constantly on that writes smaller amounts of
data to many CFs, and does some reads.

The total number of CFs are 114, but quite a few of them are not used.

Thanks,
Arindam

[1] [14/10/2013:19:15:08 PDT] ScheduledTasks:1:  WARN GCInspector.java (line 145) Heap is
0.8287082580489245 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will
now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this automatically

[2] Object Histogram:

num       #instances    #bytes  Class description
--------------------------------------------------------------------------
1:              152855  86035312        int[]
2:              13395   45388008        long[]
3:              49517   9712000 java.lang.Object[]
4:              120094  8415560 char[]
5:              145106  6965088 java.nio.HeapByteBuffer
6:              40525   5891040 * ConstMethodKlass
7:              231258  5550192 java.lang.Long
8:              40525   5521592 * MethodKlass
9:              134574  5382960 java.math.BigInteger
10:             36692   4403040 java.net.SocksSocketImpl
11:             3741    4385048 * ConstantPoolKlass
12:             63875   3538128 * SymbolKlass
13:             104048  3329536 java.lang.String
14:             132636  3183264 org.apache.cassandra.db.DecoratedKey
15:             97466   3118912 java.util.concurrent.ConcurrentHashMap$HashEntry
16:             97216   3110912 com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node


Mime
View raw message