Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of abarua@247-inc.com designates
 213.199.154.77 as permitted sender)
From: Arindam Barua <abarua@247-inc.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: RE: Heap almost full
Thread-Topic: Heap almost full
Thread-Index: Ac7KA6yqOlnSyN9gSGW9yuobJHk32QHtTm+AAHzq7wAAvWh7AA==
Date: Fri, 1 Nov 2013 01:59:49 +0000
Message-ID: 
 <e09f047d288740148a7f83029781c330@HKNPR03MB146.apcprd03.prod.outlook.com>
References: 
 <ae5e7091c7a94ea2b14a3a23744d235b@HKNPR03MB146.apcprd03.prod.outlook.com>
 <CA+VSrLp0ynRqsQyM00e094-=qC3fZb=Vs6OgiRVfGmpq+iD=zw@mail.gmail.com>
 <C876772D-C9FD-46D6-B8BA-E06D68F17FB3@thelastpickle.com>
In-Reply-To: <C876772D-C9FD-46D6-B8BA-E06D68F17FB3@thelastpickle.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


Thank you for your responses. In another recent test, the heap actually got=
 full, and we got an out of memory error. When we analyzed the heap, almost=
 all of it was memtables. Is there any known issue with 1.1.5 which causes =
memtable_total_space_in_mb not to be respected, or not defaulting to 1/3rd =
of the heap size? Or is it possible that the load in the test is that high =
that Cassandra is not able to keep flushing even though it starts the proce=
ss when memtable_total_space_in_mb is 1/3rd of the heap?

We recently switched to LeveledCompaction, however, when we got the earlier=
 heap warning, that was running on SizeTiered.
The latest test was running on high performance 32-core, 128 GB RAM, 7 RAID=
-0 1TB disks (regular). Earlier tests were run on lesser hardware with the =
same load, but there was no memory problem. We are running more tests to ch=
eck if this is always reproducible.

Answering some of the earlier questions if it helps:

We have Cassandra 1.1.5 running in production. Upgrading to the latest 1.2.=
x release is on the roadmap, but till then this needs to be figured out.

> - How many data do you got per node ?
We are running into these errors while running tests in QA starting with 0 =
load. These are around 4 hr tests which end up adding under 1 GB of data on=
 each node of a 4-node ring, or a 2-node ring.

> - What is the value of the "index_intval" (cassandra.yaml) ?
It's the default value of 128.

Thanks,
Arindam

-----Original Message-----
From: Aaron Morton [mailto:aaron@thelastpickle.com]=20
Sent: Monday, October 28, 2013 12:09 AM
To: Cassandra User
Subject: Re: Heap almost full

> 1] [14/10/2013:19:15:08 PDT] ScheduledTasks:1:  WARN GCInspector.java (li=
ne 145) Heap is 0.8287082580489245 full.  You may need to reduce memtable a=
nd/or cache sizes.  Cassandra will now flush up to the two largest memtable=
s to free up memory.  Adjust flush_largest_memtables_at threshold in cassan=
dra.yaml if you don't want Cassandra to do this automatically
This means that the CMS GC was unable to free memory quickly, you've not ru=
n out but may do under heavy load.=20

CMS uses CPU resources to do it's job, how much CPU do you have available ?=
=20
To check the behaviour of the CMS collector using JConsole or another tool =
to watch the heap size, you should see a nice saw tooth graph. It should gr=
adually grow then drop quickly to below 3ish GB. If the size of CMS is not =
low enough you will spend more time in GC.=20

You may also want to adjust flush_largest_memtables_at to be .8 to give CMS=
 a chance to do it's work. It starts at .75

> In 1.2+ bloomfilters are off-heap, you can use vnodes...
+1 for 1.2 with off heap bloom filters.=20

> - increasing the heap to 10GB.

-1=20
Unless you have a node under heavy memory problems, pre 1.2 with 1+billion =
rows and lots of bloom filters, increasing the heap is not the answer. It w=
ill increase the time taken for ParNew CMS and in kicks the problem down th=
e road.=20

Cheers
=20
-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 26/10/2013, at 8:32 am, Alain RODRIGUEZ <arodrime@gmail.com> wrote:

> If you are starting with Cassandra I really advice you to start with 1.2.=
11
>=20
> In 1.2+ bloomfilters are off-heap, you can use vnodes...
>=20
> "I summed up the bloom filter usage reported by nodetool cfstats in all t=
he CFs and it was under 50 MB."
>=20
> This is quite a small value. Is there no error in your conversion from By=
tes read in cfstats ?
>=20
> If you are trying to understand this could you tell us :
>=20
> - How many data do you got per node ?
> - What is the value of the "index_intval" (cassandra.yaml) ?
>=20
> If you are trying to fix this, you can try :
>=20
> - changing the "memtable_total_space_in_mb" to 1024
> - increasing the heap to 10GB.
>=20
> Hope this will help somehow :).
>=20
> Good luck
>=20
>=20
> 2013/10/16 Arindam Barua <abarua@247-inc.com>
> =20
>=20
> During performance testing being run on our 4 node Cassandra 1.1.5 cluste=
r, we are seeing warning logs about the heap being almost full - [1]. I'm t=
rying to figure out why, and how to prevent it.
>=20
> =20
>=20
> The tests are being run on a Cassandra ring consisting of 4 dedicated box=
es with 32 GB of RAM each.
>=20
> The heap size is set to 8 GB as recommended.
>=20
> All the other relevant settings I know off are the default ones:
>=20
> -          memtable_total_space_in_mb is not set in the yaml, so should d=
efault to 1/3rd the heap size.
>=20
> -          They key cache should be 100 MB at the most. I checked the key=
 cache the day after the tests were run via nodetool info, and it reported =
4.5 MB being used.
>=20
> -          row cache is not being used
>=20
> -          I summed up the bloom filter usage reported by nodetool cfstat=
s in all the CFs and it was under 50 MB.
>=20
> =20
>=20
> The resident size of the cassandra process accd to top is 8.4g even now. =
Did a heap histogram using jmap, but not sure how to interpret those result=
s usefully - [2].
>=20
> =20
>=20
> Performance test details:
>=20
> -          The test is write only, and is writing relatively large amount=
 of data to one CF.
>=20
> -          There is some other traffic that is constantly on that writes =
smaller amounts of data to many CFs, and does some reads.
>=20
> =20
>=20
> The total number of CFs are 114, but quite a few of them are not used.
>=20
> =20
>=20
> Thanks,
>=20
> Arindam
>=20
> =20
>=20
> [1] [14/10/2013:19:15:08 PDT] ScheduledTasks:1:  WARN GCInspector.java (l=
ine 145) Heap is 0.8287082580489245 full.  You may need to reduce memtable =
and/or cache sizes.  Cassandra will now flush up to the two largest memtabl=
es to free up memory.  Adjust flush_largest_memtables_at threshold in cassa=
ndra.yaml if you don't want Cassandra to do this automatically
>=20
> =20
>=20
> [2] Object Histogram:
>=20
> =20
>=20
> num       #instances    #bytes  Class description
>=20
> -------------------------------------------------------------------------=
-
>=20
> 1:              152855  86035312        int[]
>=20
> 2:              13395   45388008        long[]
>=20
> 3:              49517   9712000 java.lang.Object[]
>=20
> 4:              120094  8415560 char[]
>=20
> 5:              145106  6965088 java.nio.HeapByteBuffer
>=20
> 6:              40525   5891040 * ConstMethodKlass
>=20
> 7:              231258  5550192 java.lang.Long
>=20
> 8:              40525   5521592 * MethodKlass
>=20
> 9:              134574  5382960 java.math.BigInteger
>=20
> 10:             36692   4403040 java.net.SocksSocketImpl
>=20
> 11:             3741    4385048 * ConstantPoolKlass
>=20
> 12:             63875   3538128 * SymbolKlass
>=20
> 13:             104048  3329536 java.lang.String
>=20
> 14:             132636  3183264 org.apache.cassandra.db.DecoratedKey
>=20
> 15:             97466   3118912 java.util.concurrent.ConcurrentHashMap$Ha=
shEntry
>=20
> 16:             97216   3110912 com.googlecode.concurrentlinkedhashmap.Co=
ncurrentLinkedHashMap$Node
>=20
> =20
>=20
>=20
>=20