zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andor Molnar <an...@cloudera.com>
Subject Re: Observers: a link to message bus?
Date Thu, 25 Jan 2018 11:50:08 GMT
You're pretty much right I believe. There's a new feature in 3.5 called TTL
which could help removing old nodes, but I can't tell when it's going to be
stable.

Let me recommend you the ZK book: O'Reilly: Zookeeper - Distributed Process
Coordination: Chapter 4: Dealing with State Change.
The entire chapter is talking about how to deal with the problem you're
facing and what can you do about it with Zookeeper.

Regards,
Andor



On Wed, Jan 24, 2018 at 2:59 PM, Bruno Freudensprung <
bfreudensprung@expertsystem.com> wrote:

> Hi Andor,
>
> Thanks a lot for your answer!
> I have another question if you don't mind. Once again I am just curious
> about Zookeeper and I don't want to waste your time, so please simply
> ignore this message if it is the case.
> Hacking the ObserverZooKeeperServer (hooking custom code when INFORM
> messages are commited) would enable a solution based on changing the value
> of the same zk node, over and over. In that case the number of zk nodes
> will remain small, even if "change throughput" is high. However a solution
> based on observer+client will have to be based on adding new (ephemeral)
> nodes to the same zk node (otherwise the client might miss some updates).
> So I'll have to find a way to "timeout" and remove those nodes at some
> point. Does that sound correct?
> Regards,
>
> Bruno.
>
> -----Message d'origine-----
> De : Andor Molnar [mailto:andor@cloudera.com]
> Envoyé : mercredi 24 janvier 2018 11:16
> À : user@zookeeper.apache.org
> Objet : Re: Observers: a link to message bus?
>
> Hi Bruno,
>
> Unfortunately I have no idea what plugin system does the author of this
> doc refer to, but I can think of the following 2 solutions:
>
> 1) Fire up some observers in ZK cluster and connect to them via some
> standard client. Create a zk node and create a watch for its children.
> Producer can create nodes as children of this special node and connected
> clients (consumers) can receive them in an ordered manner. This setup will
> not affect the overall performance of the cluster, because the 'message
> bus' only uses observers to achieve its goal.
>
> 2) Write your custom Observer node by implementing ZK protocol that can
> receive and parse INFORM messages. These messages are consumed by Observers
> too. Obviously this solution could be enormously cumbersome, although
> Observers are the least complicated nodes of a ZK ensemble, but I'm not
> aware of any existing 'plugin' solution which would achieve similar
> forwarder mechanism.
>
> Regards,
> Andor
>
>
> On Wed, Jan 17, 2018 at 2:57 PM, Martin Gainty <mgainty@hotmail.com>
> wrote:
>
> >
> >
> > ________________________________
> > From: Bruno Freudensprung <bfreudensprung@expertsystem.com>
> > Sent: Tuesday, January 16, 2018 7:54 AM
> > To: user@zookeeper.apache.org
> > Subject: Observers: a link to message bus?
> >
> > Hi,
> >
> > I have a question regarding this part of the Observers Guide (
> > https://zookeeper.apache.org/doc/r3.4.11/zookeeperObservers.html):
> > ZooKeeper Observers<https://zookeeper.apache.org/doc/r3.4.11/
> > zookeeperObservers.html>
> > zookeeper.apache.org
> > Setting up a ZooKeeper ensemble that uses Observers is very simple,
> > and requires just two changes to your config files. Firstly, in the
> > config file of every node ...
> >
> >
> >
> > "As a link to a message bus : [...] Observers would give a natural
> > integration point for this work: a plug-in mechanism could be used to
> > attach the stream of proposals an Observer sees to a publish-subscribe
> > system"
> >
> > I've not been able to find such a plug-in mechanism in the source code
> > of ZooKeeper. For the moment the only integration point I can think of
> > is the commitRequest method of the
> > org.apache.zookeeper.server.quorum.ObserverZooKeeperServer
> > class. Am I looking in the right direction?
> >
> > To be honest: this is for the moment only pure curiosity since Apache
> > Kafka seems to be a more "natural" and out-of-the-box option for an
> > ordered message bus. However since I might probably need ZooKeeper for
> > its distributed lock and leader election features, I was wondering if
> > I could use it as an ordered message bus as well.
> > MG>if ActiveMQ does indeed support ordered messaging for multiple
> > MG>active
> > consumers thru Message Groups as stated:
> > MG>Message Groups which is the way to load balance multiple active
> > consumers listening to the same queue while preserving message order.
> > MG>i have asked why does Zookeeper not support
> > MG>MessageOrderingTechnology
> > such as ActiveMQ  as a plugin?
> > MG>have you discovered "plugin integration" within the Zookeeper
> > Architecture?
> >
> > Best regards,
> >
> > Bruno.
> > MG>Thanks Bruno
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message