Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAF+A=7pZ2FC6pOzBwN=BDHQcgBiRk2uAX1334e800ugOBn4fcQ@mail.gmail.com>
References: 
 <CAF+A=7rfiwAP-sAGUMGRzGF15cLA4D9cQmP=Ra_XUUCFU_ceJA@mail.gmail.com>
 <CANykdua1XfTL-351ZtJZfP8Z0oO-dfec=P9CKjnz5oXF1FRYkA@mail.gmail.com>
 <CAF+A=7p1hZtfax4_Gu32r4a-Q3pmtrrSP47RKAOmU2QX_Sp2ZQ@mail.gmail.com>
 <CANykduZorvbfgxquJvNeE+tnXCz6XJa529FZ0VYczo+4qiNWJg@mail.gmail.com>
 <CAF+A=7pZ2FC6pOzBwN=BDHQcgBiRk2uAX1334e800ugOBn4fcQ@mail.gmail.com>
From: Check Peck <comptechgeeky@gmail.com>
Date: Sun, 13 Sep 2015 14:04:28 -0700
Message-ID: 
 <CAF+A=7o4oJ9x-c=N+=adJOMaiRM0YPZhyLRFu410RRRoZ575Kw@mail.gmail.com>
Subject: Re: zookeeper design problem
To: user <user@zookeeper.apache.org>
Content-Type: multipart/alternative; boundary=001a113475ae9640a6051fa74cc5

--001a113475ae9640a6051fa74cc5
Content-Type: text/plain; charset=UTF-8

Sorry my use case will be like this:

In my case, what does it mean? If I have 1000 clients and each client has
100 events znodes and those znodes 100 more childs and those childs have
some data in it. Then what will fail if I try to do something?

If the length of all childrens znode names is greater than 1 MB, then it
will fail right? Data can be more than 1MB in each znode correct?

This is what I am trying to understand basis on zookeeper limitation.

On Sun, Sep 13, 2015 at 1:47 PM, Check Peck <comptechgeeky@gmail.com> wrote:

> Ok understood. As per other thread in which Jordan mention to me this:
>
>
>
> *In terms of length, ZooKeeper has a 1MB limit per API call. So, for
> example, if you call getChildren(), the entire length of all the children
> can't exceed 1MB (unless you reconfigure ZK).*
> In my case, what does it mean? If I have 1000 clients and each client has
> 100 znodes and those znodes have some data in it. Then what will fail if I
> try to do something?
>
> I am trying to get a better idea on this.
>
> On Sat, Sep 12, 2015 at 11:22 PM, Cameron McKenzie <mckenzie.cam@gmail.com
> > wrote:
>
>> You can call the event / timestamp zNode's whatever you like. I'd just use
>> a persistent sequential node for the name, and have your reaper make its
>> decisions based on the creation / modification time of the event /
>> timestamp zNode that you're referring to rather than the name of the zNode
>> itself.
>>
>> On Sun, Sep 13, 2015 at 4:03 PM, Check Peck <comptechgeeky@gmail.com>
>> wrote:
>>
>> > Thanks Cameron for suggestion. Is it ok if I have timestamp1 instead of
>> > event1, timestamp2 instead of event2 and so on? Or do you think it
>> might be
>> > bad idea.
>> >
>> > This timestamp will be System.currentTimeMillis() from Java.
>> >
>> > On Sat, Sep 12, 2015 at 10:06 PM, Cameron McKenzie <
>> mckenzie.cam@gmail.com
>> > >
>> > wrote:
>> >
>> > > Can't you just add a zNode under client_xxx for each of your 'event'
>> > > triplets, and then have the reaper go and remove that whole child node
>> > when
>> > > the 'event' node is beyond a certain age? Something like:
>> > >
>> > > /root/clients/client_100/event1/metric
>> > > /root/clients/client_100/event1/transaction
>> > > /root/clients/client_100/event1/log
>> > > /root/clients/client_100/event2/metric
>> > > /root/clients/client_100/event2/transaction
>> > > /root/clients/client_100/event2/log
>> > > /root/clients/client_101/event1/metric
>> > > /root/clients/client_101/event1/transaction
>> > > /root/clients/client_101/event1/log
>> > >
>> > >
>> > > On Sun, Sep 13, 2015 at 7:59 AM, Check Peck <comptechgeeky@gmail.com>
>> > > wrote:
>> > >
>> > > > I am working on designing the zookeeper node hierarchy so looking
>> for
>> > > some
>> > > > suggestion. I have a clients node under which I will have multiple
>> > > clients
>> > > > (for ex: client_100, client_101)
>> > > >
>> > > >     /root/clients/client_100
>> > > >     /root/clients/client_101
>> > > >     /root/clients/client_102
>> > > >
>> > > > Now inside each client, I need to store three things:
>> > > >
>> > > >    1. First is metric data which will be in JSON format (max 15
>> > key-value
>> > > >    pairs).
>> > > >    2. Second is transaction data which will be plain simple string
>> > (this
>> > > >    data should be less than 1 MB).
>> > > >    3. Third is client logs data which will also be plain simple
>> string
>> > > >    (this data should also be less than 1 MB).
>> > > >
>> > > > Also now most important thing, I can have multiple of above three
>> > things
>> > > > for each client. Meaning for client_100, I can have two metric data,
>> > two
>> > > > transaction data and two client logs. Here two is just a number it
>> can
>> > > be X
>> > > > but I will have my reaper process running that can clean the old
>> > "metric
>> > > > data, transaction data and client logs" for each client after
>> certain
>> > > > period or some threshold.
>> > > >
>> > > > In short, for each client I have a list of above three things which
>> I
>> > > need
>> > > > to show on zookeeper. What is the right design for this so that I
>> don't
>> > > > destroy zookeeper. I can have each znode for each of above three
>> things
>> > > but
>> > > > I am not sure how can I bucket this since it will be multiple for
>> each
>> > > > clients.
>> > > >
>> > > > Also max number of clients I can have is 1000 that's all. But I will
>> > have
>> > > > another reaper process for this as well to keep deleting older
>> clients
>> > > > which doesn't had any activity from a long time.
>> > > >
>> > >
>> >
>>
>
>

--001a113475ae9640a6051fa74cc5--