zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: Question regarding the maximum number of ZNODES a zookeeper
Date Wed, 09 Nov 2011 22:31:09 GMT
On Wed, Nov 9, 2011 at 1:14 PM, Aniket Chakrabarti
<chakraba@cse.ohio-state.edu> wrote:
> I am trying to load a huge matrix(100,000 by 500) to my zookeeper instance.
> Each element of the matrix is a ZNODE and value of each element is a
> digit(0-9).
>
> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing an
> error after that. Mostly it is throwing a "-5" error code which is a
> marshalling/unmarshalling error. I'm using the perl interface of the
> zookeeper.
>
> My question is: Is there a limit to the maximum number of ZNODES a zookeeper
> instance can hold or this is limited by the system memory?
>
> Any pointers on how to avoid the error would be very helpful.

Available heap memory is really the only limit. Try

$ JVMFLAGS=-D-Xmx<mem in gig>g bin/zkServer.sh

also

$ sudo jmap -heap <jvm pid>

will give you some insight into whether it was set correctly or not
(ie MaxHeapSize)

The most I've tried is 5 million znodes with 25 million watches (using
zkpython, but zkperl should be fine). iirc that was 8gig heap, but
ymmv depending on data size.

You may also need to tune the GC at some point (would suggest turning
on cms and parallel collector) to limit stop the world pauses.

Regards,

Patrick

Mime
View raw message