hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chang Song <tru64...@me.com>
Subject Re: Persistent watch stream?
Date Sat, 13 Nov 2010 07:08:17 GMT

Thank you, Ted.

As far as getChildren goes, you are right.
Since you will get the latest children change and at the same time set a watch,
you will definitely not lose any children node change.

Thank you.

Chang


Nov 13, 2010, 12:51 PM, Ted Dunning 작성:

> Persistent watches were omitted from ZK on purpose because of the perceived
> danger of not have a load shedding mechanism.
> 
> Note that when you get a notification, the query you do to get the next
> state typically sets the next watch.  This guarantees that you don't lose
> anything, but it may mean that you effectively get notified of multiple
> changes in one notification.
> 
> If you really, really want all notifications in order, then what you are
> really looking for is a kind of a distributed transaction log.  For small
> applications, you can implement this by writing logs into ZK znodes.  Your
> client should remember where they were and as they are notified, they read
> the current file to catch up.  This has the downside that to update a file
> you have to completely rewrite it which makes it inconvenient to put a bunch
> of stuff into a single chunk of the log.  You would also need a watcher on
> the directory to notify you when new log files are created.  Aside from the
> slightly quadratic update problem, this does what you need.
> 
> You can also check out Bookkeeper which is a more scalable distributed
> transaction logging system.  It addresses the complications of (mis) using
> ZK as a log system and attempts to give you a reliable and robust
> transaction log.
> 
> On Fri, Nov 12, 2010 at 7:16 PM, Chang Song <tru64ufs@me.com> wrote:
> 
>> 
>> Hi.
>> 
>> It's been a couple of weeks since we started integrating zookeeper into our
>> app.
>> Great piece of software. Cut down LOC by more than half.
>> Thank you for open sourcing Zookeeper.
>> 
>> I have a couple of questions, though.
>> I was wondering you have any plan to support persistent watch stream
>> feature?
>> 
>> One-time watch is very inconvenient for us.
>> We need to get a stream of notifications in order. We are talking thousands
>> of clients.
>> Since notification can happen in bulk, we need to set a watch first, and
>> once we get a callback,
>> we need to check periodically what happens to the children of the watch
>> nodes.
>> 
>> You can consider this as a automatic server-side watch registration
>> feature: I'd call this sticky watch.
>> I think it is easier to implement this as a sticky watch.
>> 
>> Another question is if it is possible to coalesce predefined number of
>> changes for watch callbacks.
>> We have observed that if there are many changes in children node, clients
>> gets different number of
>> messages in bulks. This makes the architecture of our application from
>> event-driven to polling for a while.
>> Thus pretty much the same reasoning behind the first question.
>> 
>> Any comments welcome.
>> 
>> Thank you.
>> 
>> Chang
>> 
>> 
>> 
>> 


Mime
View raw message