zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: Question regarding the maximum number of ZNODES a zookeeper
Date Wed, 09 Nov 2011 23:00:37 GMT
er, make that JVMFLAGS=-Xmx<mem in gig>g bin/zkServer.sh

(no -D)

Patrick

On Wed, Nov 9, 2011 at 2:31 PM, Patrick Hunt <phunt@apache.org> wrote:
> On Wed, Nov 9, 2011 at 1:14 PM, Aniket Chakrabarti
> <chakraba@cse.ohio-state.edu> wrote:
>> I am trying to load a huge matrix(100,000 by 500) to my zookeeper instance.
>> Each element of the matrix is a ZNODE and value of each element is a
>> digit(0-9).
>>
>> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing an
>> error after that. Mostly it is throwing a "-5" error code which is a
>> marshalling/unmarshalling error. I'm using the perl interface of the
>> zookeeper.
>>
>> My question is: Is there a limit to the maximum number of ZNODES a zookeeper
>> instance can hold or this is limited by the system memory?
>>
>> Any pointers on how to avoid the error would be very helpful.
>
> Available heap memory is really the only limit. Try
>
> $ JVMFLAGS=-D-Xmx<mem in gig>g bin/zkServer.sh
>
> also
>
> $ sudo jmap -heap <jvm pid>
>
> will give you some insight into whether it was set correctly or not
> (ie MaxHeapSize)
>
> The most I've tried is 5 million znodes with 25 million watches (using
> zkpython, but zkperl should be fine). iirc that was 8gig heap, but
> ymmv depending on data size.
>
> You may also need to tune the GC at some point (would suggest turning
> on cms and parallel collector) to limit stop the world pauses.
>
> Regards,
>
> Patrick
>

Mime
View raw message