zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szekeres, Zoltan" <Zoltan.Szeke...@morganstanley.com>
Subject RE: Use-case with lots of child nodes
Date Fri, 03 Jun 2016 16:48:26 GMT
Thanks Patrick for the response. We’ll make use of the system property if needed.

What’s not clear for me from the jira, I’m not sure if I’ve missed something, is the
reason for the asymmetry in the default allowed length of the incoming message on client side
and on server side. It is 4MB on the client side and 1MB on the server side. Is the asymmetry
intentional or accidental?

Thanks,
Zoltan Szekeres

From: Patrick Hunt [mailto:phunt1@gmail.com]
Sent: Thursday, June 02, 2016 12:02 AM
To: UserZooKeeper
Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure); Szekeres, Zoltan (IST)
Subject: Re: Use-case with lots of child nodes


On Thu, May 26, 2016 at 2:22 AM, Szekeres, Zoltan <Zoltan.Szekeres@morganstanley.com<mailto:Zoltan.Szekeres@morganstanley.com>>
wrote:
Thanks for the response.
I'm trying to understand what operational stability issues would there be. I'm also planning
to do a latency test for requests at the time I'm requesting getChildren on "/someparentNode".

To give more detail I have a primary and secondary use-case:
My primary use-case includes having watchers on the children of "/someparentNode" and requesting
getChildren for "/someparentNode/LotsOfChildNodesHere-N" (which only has a couple child nodes).
My secondary use-case would be requesting the children of "/someparentNode", which would be
only occasionally for reporting purposes (which has a lot of child nodes and probably won't
be as much as 70k nodes, but I hit the limit there).

I'm looking for answers for the following questions:
What are the stability issues that you think might occur having lots of nodes under one node,
even if we read them rarely?
Can I reliable use the "jute.maxbuffer" system property on the client in the future?

I don't see why not. We have it there as a safety mechanism and to corral folks towards using
it "the right way". Although opinions on what "the right way" means may differ, which is probably
why it's configurable. ;-)

Looking for answers whether the asymmetry of the default value on client side and on server
side is accidental or intentional.

See this jira, it's currently being discussed: https://issues.apache.org/jira/browse/ZOOKEEPER-2430

Patrick


Any advice is much appreciated.

Thanks,
Zoltan Szekeres

-----Original Message-----
From: Tom Crayford [mailto:tcrayford@heroku.com<mailto:tcrayford@heroku.com>]
Sent: Wednesday, May 25, 2016 12:06 PM
To: user@zookeeper.apache.org<mailto:user@zookeeper.apache.org>
Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
Subject: Re: Use-case with lots of child nodes

Hi,

I'd recommend rethinking your use case. Zookeeper isn't really great with large amounts of
data, nor does it handle high volumes of changes that well.

It looks like setting that system property will work for now, but I'd expect trying to use
such a high volume of child nodes would have severe consequences for operational stability.

Thanks

Tom Crayford
Heroku Kafka

On Wednesday, 25 May 2016, Szekeres, Zoltan < Zoltan.Szekeres@morganstanley.com<mailto:Zoltan.Szekeres@morganstanley.com>>
wrote:

> Hi ZooKeeper users,
>
> I have a use-case where I need to create a very large number (~70,000)
> of child nodes under a parent. These nodes themselves contain no data
> and will only have a handful of child nodes themselves.
> e.g.
>  /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel
> /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel
> ...
> /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLevel
>
> I've read
> (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html)
> there is a limit of 1 MB. But I hit the limit for the getChildren
> operation around 4 MB. I'm interested in what's causing the difference in the limit.
> I was able to increase the 4MB limit by setting the "jute.maxbuffer"
> system property at the client. Can I reliably use this system property
> in the future to set the limit?
>
> Any advice is appreciated.
>
> Thank you,
> Zoltan Szekeres
>
>
>
> ________________________________
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall Street Reform and Consumer Protection Act. If you have received
> this communication in error, please destroy all electronic and paper
> copies; do not disclose, use or act upon the information; and notify
> the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to
> the extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers If you cannot access these
> links, please notify us by reply message and we will send the contents
> to you. By messaging with Morgan Stanley you consent to the foregoing.
>

--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained
herein are not intended to be, and do not constitute, advice within the meaning of Section
975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received
this communication in error, please destroy all electronic and paper copies; do not disclose,
use or act upon the information; and notify the sender immediately. Mistransmission is not
intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the
extent permitted under applicable law, to monitor electronic communications. This message
is subject to terms available at the following link: http://www.morganstanley.com/disclaimers.
If you cannot access these links, please notify us by reply message and we will send the contents
to you. By messaging with Morgan Stanley you consent to the foregoing.



________________________________

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained
herein are not intended to be, and do not constitute, advice within the meaning of Section
975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received
this communication in error, please destroy all electronic and paper copies; do not disclose,
use or act upon the information; and notify the sender immediately. Mistransmission is not
intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the
extent permitted under applicable law, to monitor electronic communications. This message
is subject to terms available at the following link: http://www.morganstanley.com/disclaimers
If you cannot access these links, please notify us by reply message and we will send the contents
to you. By messaging with Morgan Stanley you consent to the foregoing.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message