zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <phu...@gmail.com>
Subject Re: Use-case with lots of child nodes
Date Wed, 01 Jun 2016 22:02:25 GMT
On Thu, May 26, 2016 at 2:22 AM, Szekeres, Zoltan <
Zoltan.Szekeres@morganstanley.com> wrote:

> Thanks for the response.
> I'm trying to understand what operational stability issues would there be.
> I'm also planning to do a latency test for requests at the time I'm
> requesting getChildren on "/someparentNode".
>
> To give more detail I have a primary and secondary use-case:
> My primary use-case includes having watchers on the children of
> "/someparentNode" and requesting getChildren for
> "/someparentNode/LotsOfChildNodesHere-N" (which only has a couple child
> nodes).
> My secondary use-case would be requesting the children of
> "/someparentNode", which would be only occasionally for reporting purposes
> (which has a lot of child nodes and probably won't be as much as 70k nodes,
> but I hit the limit there).
>
> I'm looking for answers for the following questions:
> What are the stability issues that you think might occur having lots of
> nodes under one node, even if we read them rarely?
> Can I reliable use the "jute.maxbuffer" system property on the client in
> the future?
>

I don't see why not. We have it there as a safety mechanism and to corral
folks towards using it "the right way". Although opinions on what "the
right way" means may differ, which is probably why it's configurable. ;-)


> Looking for answers whether the asymmetry of the default value on client
> side and on server side is accidental or intentional.
>

See this jira, it's currently being discussed:
https://issues.apache.org/jira/browse/ZOOKEEPER-2430

Patrick


>
> Any advice is much appreciated.
>
> Thanks,
> Zoltan Szekeres
>
> -----Original Message-----
> From: Tom Crayford [mailto:tcrayford@heroku.com]
> Sent: Wednesday, May 25, 2016 12:06 PM
> To: user@zookeeper.apache.org
> Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
> Subject: Re: Use-case with lots of child nodes
>
> Hi,
>
> I'd recommend rethinking your use case. Zookeeper isn't really great with
> large amounts of data, nor does it handle high volumes of changes that well.
>
> It looks like setting that system property will work for now, but I'd
> expect trying to use such a high volume of child nodes would have severe
> consequences for operational stability.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Wednesday, 25 May 2016, Szekeres, Zoltan <
> Zoltan.Szekeres@morganstanley.com> wrote:
>
> > Hi ZooKeeper users,
> >
> > I have a use-case where I need to create a very large number (~70,000)
> > of child nodes under a parent. These nodes themselves contain no data
> > and will only have a handful of child nodes themselves.
> > e.g.
> >  /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel
> > /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel
> > ...
> > /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLevel
> >
> > I've read
> > (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html)
> > there is a limit of 1 MB. But I hit the limit for the getChildren
> > operation around 4 MB. I'm interested in what's causing the difference
> in the limit.
> > I was able to increase the 4MB limit by setting the "jute.maxbuffer"
> > system property at the client. Can I reliably use this system property
> > in the future to set the limit?
> >
> > Any advice is appreciated.
> >
> > Thank you,
> > Zoltan Szekeres
> >
> >
> >
> > ________________________________
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> > Wall Street Reform and Consumer Protection Act. If you have received
> > this communication in error, please destroy all electronic and paper
> > copies; do not disclose, use or act upon the information; and notify
> > the sender immediately. Mistransmission is not intended to waive
> > confidentiality or privilege. Morgan Stanley reserves the right, to
> > the extent permitted under applicable law, to monitor electronic
> > communications. This message is subject to terms available at the
> following link:
> > http://www.morganstanley.com/disclaimers If you cannot access these
> > links, please notify us by reply message and we will send the contents
> > to you. By messaging with Morgan Stanley you consent to the foregoing.
> >
>
>
>
> --------------------------------------------------------------------------------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies; do
> not disclose, use or act upon the information; and notify the sender
> immediately. Mistransmission is not intended to waive confidentiality or
> privilege. Morgan Stanley reserves the right, to the extent permitted under
> applicable law, to monitor electronic communications. This message is
> subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers. If you cannot access these
> links, please notify us by reply message and we will send the contents to
> you. By messaging with Morgan Stanley you consent to the foregoing.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message