Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (athena.apache.org: message received from 54.191.145.13
 which is an MX secondary for user@zookeeper.apache.org)
From: Karol Dudzinski <karoldudzinski@gmail.com>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Subject: Re: OutOfMemory Error
Message-Id: <CC795230-BF64-474D-87E2-36EA72FCE374@gmail.com>
Date: Fri, 24 Apr 2015 19:09:10 +0100
References: 
 <CA+TiGKpwexMVdBnaRk12d8yWgxONJd7N+OdmRSUs7E9ELxPWFg@mail.gmail.com>
In-Reply-To: 
 <CA+TiGKpwexMVdBnaRk12d8yWgxONJd7N+OdmRSUs7E9ELxPWFg@mail.gmail.com>
To: "user@zookeeper.apache.org" <user@zookeeper.apache.org>

Hi,

Do you know if any of the services that use your ZK create ACLs that are pot=
entially unique and one-time-ish?  I recently hit a similar problem and disc=
overed that the DataTree has an ACL cache that never gets anything removed f=
rom it.  That was by far and away the largest memory consumer I found when a=
nalysing the heap dump.  If this is the case then you should see lots of ACL=
 objects on the heap.

I filed a JIRA for this and keep meaning to submit a patch but sadly haven't=
 got round to it.  As an interim solution, I wrote a tool which uses the Dat=
aTree class and the serialisation utils to purge this cache of unused entrie=
s.  I my case it shrank the snapshot from 500MB to 12MB!  The time to write t=
he snapshot went from 40 seconds to less than 1 second as a result.

Thanks,
Karol


> On 24 Apr 2015, at 18:45, CP Mishra <mishracp@gmail.com> wrote:
>=20
> Hi,
>=20
> I am running a 3 node ZK ensemble on 3 VMs (2 CPU, 32GB RAM) in the test
> environment. Lately, I have been getting OutOfMemoryError on all three ZK
> nodes. ZK has been configured with 6GB heap size. The same ZK ensemble is
> shared between Kafka, HDFS HA and another custom service.
>=20
> I analyzed the heap dump and 5.8+ GB is being used by DataTree.  I don't
> have a purge policy in place and size of ZK data directory stands at ~14 G=
B
> now.  There is enough space on the disk holding ZK data (20% used).
>=20
> As soon as I restart a ZK node, it grows to use all 6GB and starts Full GC=

> every 1-2 sec. In 3-5 minutes, it throws OOM: GC Overhead exceeded.
>=20
> I would appreciate any help in diagnosing the issue.
>=20
> Thanks,
> CP Mishra