Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAFcPz7h0W0qtmjMZYe034mJza0S6NebvGG5TfgYSO3Km6ZZDDA@mail.gmail.com>
References: <alpine.LRH.2.02.1202081713090.7232@pyramidlake.corp.yahoo.com>
	<CAKF1A=vmvGAeCpHp9Nha+hYjJUiQEgG8LJ6yCgEgiE9zKPKxNg@mail.gmail.com>
	<CAFcPz7h0W0qtmjMZYe034mJza0S6NebvGG5TfgYSO3Km6ZZDDA@mail.gmail.com>
Date: Thu, 9 Feb 2012 10:17:12 -0800
Message-ID: 
 <CALF7aMSHzUMACmXf6UrRvB5sFXC9aM02uRTo984NefMJoF4oQw@mail.gmail.com>
Subject: Re: ZooKeeper Memory Usage
From: Mahadev Konar <mahadev@hortonworks.com>
To: user@zookeeper.apache.org
Cc: Mike Schilli <m@perlmeister.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

This is interesting and important.

Cesar, what jvm options are you running with? Can you the options in:

https://cwiki.apache.org/confluence/display/ZOOKEEPER/Troubleshooting

Atleast get the GC logs that we can look at?

This will be very interesting.

mahadev


2012/2/9 C=E9sar =C1lvarez N=FA=F1ez <cesar.alvarez.nunez@gmail.com>:
> In my case, our stress test show up a linear increase of "tenured memory"
> from 0 to > 3GiB with ZK 3.4.0 whereas the same stress-test with 3.3.3
> keeps "tenured memory" stable and < 10MiB.
>
> The stress test performs many zNodes creation and delete but the overall =
zk
> usage at any moment in time was relative small.
>
> BR,
> /C=E9sar.
>
> On Thu, Feb 9, 2012 at 3:14 PM, Camille Fournier <camille@apache.org> wro=
te:
>
>> This is really a question about how the jvm grows its heaps and resizes
>> them. If the jvm cannot allocate enough memory for the process because y=
ou
>> didn't set the max memory high enough, it will fall over. Zookeeper keep=
s
>> its entire state in memory for performance reasons, if it were to swap t=
hat
>> would be quite bad for performance.
>>
>> C
>> On Feb 8, 2012 8:23 PM, "Mike Schilli" <m@perlmeister.com> wrote:
>>
>> > We've got a ZooKeeper instance that's using about 5 GB of resident
>> > memory. Every time we restart it, it starts at 200MB, and then grows
>> > slowly until it is back at 5 GB.
>> >
>> > The large footprint is related to how much data we've got in there.
>> > What's interesting, though, is that the process size doesn't shrink if
>> > we purge some of the data.
>> >
>> > Now, this isn't a big problem, I'm just curious if the process will fa=
ll
>> > over at some point if it can't get more memory or if it'll just make d=
ue
>> > by caching less data.
>> >
>> > Also, if I remember correctly, there's a configuration variable to set
>> > the maximum size, what happens if ZK reaches that?
>> >
>> > -- -- Mike
>> >
>> > Mike Schilli
>> > m@perlmeister.com
>> >
>>


--=20
Mahadev Konar
Hortonworks Inc.
http://hortonworks.com/