zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Crayford <tcrayf...@heroku.com>
Subject Re: Use-case with lots of child nodes
Date Wed, 25 May 2016 10:06:25 GMT
Hi,

I'd recommend rethinking your use case. Zookeeper isn't really great with
large amounts of data, nor does it handle high volumes of changes that well.

It looks like setting that system property will work for now, but I'd
expect trying to use such a high volume of child nodes would have severe
consequences for operational stability.

Thanks

Tom Crayford
Heroku Kafka

On Wednesday, 25 May 2016, Szekeres, Zoltan <
Zoltan.Szekeres@morganstanley.com> wrote:

> Hi ZooKeeper users,
>
> I have a use-case where I need to create a very large number (~70,000)  of
> child nodes under a parent. These nodes themselves contain no data and will
> only have a handful of child nodes themselves.
> e.g.
>  /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel
> /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel
> ...
> /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLevel
>
> I've read (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html)
> there is a limit of 1 MB. But I hit the limit for the getChildren operation
> around 4 MB. I'm interested in what's causing the difference in the limit.
> I was able to increase the 4MB limit by setting the "jute.maxbuffer"
> system property at the client. Can I reliably use this system property in
> the future to set the limit?
>
> Any advice is appreciated.
>
> Thank you,
> Zoltan Szekeres
>
>
>
> ________________________________
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies; do
> not disclose, use or act upon the information; and notify the sender
> immediately. Mistransmission is not intended to waive confidentiality or
> privilege. Morgan Stanley reserves the right, to the extent permitted under
> applicable law, to monitor electronic communications. This message is
> subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers If you cannot access these
> links, please notify us by reply message and we will send the contents to
> you. By messaging with Morgan Stanley you consent to the foregoing.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message