zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: Question regarding the maximum number of ZNODES a zookeeper
Date Thu, 10 Nov 2011 06:18:01 GMT
Good points Ted.

Aniket, you might try using visualvm or something similar to monitor
heap usage during your test:
http://visualvm.java.net/
It will give you some insight into how quickly the heap is being used up.

You're reporting approx 4k/znode. That seems way high to me. Are you
sure that the data for each znode is small? (you initially reported it
being a digit).

I just ran a quick test using yourkit and I'm seeing more like 400
bytes per znode.

You might start bin/zkCli.sh and use the "stat" command to get more
insight into your znodes (for example the length).

Patrick

On Wed, Nov 9, 2011 at 9:07 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
> Each node can take a fair bit of overhead in memory.   That 2G you
> allocated may not go as far as you might like.  I am a bit surprised if
> this eats as much memory as your test seems to imply, but it isn't
> impossible.
>
> Have you tried a row or column compressed format?  Do you really need to
> store elements separately?
>
> Also, Zookeeper is intended primarily as a coordination service.  That
> tends to allow/encourage design decisions that may have a negative impact
> on using it as a data store.  The fact is, it tends to do much better than
> you might expect at datastore tasks, but that doesn't change the fact that
> it really isn't a great platform for that.
>
> On Wed, Nov 9, 2011 at 4:14 PM, Aniket Chakrabarti <
> chakraba@cse.ohio-state.edu> wrote:
>
>> Hi,
>>
>> I am trying to load a huge matrix(100,000 by 500) to my zookeeper
>> instance. Each element of the matrix is a ZNODE and value of each element
>> is a digit(0-9).
>>
>> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing
>> an error after that. Mostly it is throwing a "-5" error code which is a
>> marshalling/unmarshalling error. I'm using the perl interface of the
>> zookeeper.
>>
>> My question is: Is there a limit to the maximum number of ZNODES a
>> zookeeper instance can hold or this is limited by the system memory?
>>
>> Any pointers on how to avoid the error would be very helpful.
>>
>> Thanks,
>> Aniket Chakrabarti
>> PhD student
>> The Ohio State University
>>
>

Mime
View raw message