zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andor Molnar <an...@apache.org>
Subject Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Date Tue, 13 Aug 2019 19:34:52 GMT
Sure, well said. Don't get me wrong when I'm kidding Ted: I strongly
believe that Kafka is misusing ZooKeeper for their event processing,
although I don't know the details, just read the linked post on their
wiki. Details might be important here, because at first glance it
seemed to me that they expect a functionality in ZooKeeper which they
suppose to provide.

Back to the subscriber topic: I'm planning to resurrect the patch that
Jordan is referring to. 
I think it would be a great new feature in 3.6 or 4.0.

Andor


-----Original Message-----
From: Ted Dunning <ted.dunning@gmail.com>
Reply-To: dev@zookeeper.apache.org
To: dev@zookeeper.apache.org
Subject: Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata
Quorum
Date: Tue, 13 Aug 2019 12:00:36 -0700

On Tue, Aug 13, 2019 at 10:41 AM Andor Molnar <andor@apache.org> wrote:
> Hmmm... and what should Kafka do if it wants to see all events?
> UseRabbitMQ? :)

Kafka should not be using ZK to store messages.  And synchronization
ofconfiguration doesn't require message passing.
Read the original ZK paper [1] to understand that coordination and
reliablestate synchronization in ZK is not message passing. The point
is aboutproviding primitives to allow a read cache of roughly coherent
values withgood bounds on the semantics. Kafka should use ZK to
coordinate who ismaster and should use its own mechanisms to pass
messages. A key point isthe performance that can be gained by using a
cache coherence strategyrather than an "all updates" strategy.
Look at the Omega scheduler[2] paper to see the motivation behind
thesmall-steps style of scheduling and resource allocation.
Seriously, it you are doing things like designating where replicas go
andwho has the baton, it is critical to fast forward to things as they
are nowrather than things as they were. When the load hits the fan, it
isimportant to be able to ignore the water under the bridge.
This sort of topic arises in lots of other places. People complain that
itis hard to get synchronized and atomic updates to multiple files in
adistributed file system. Moral: don't use a file system when
asynchronization tool like ZK is needed.
People complain about message passing systems that they are slow in
termsof total write speed (can't write 100's of GB/s) and don't allow
native andefficient reading of state from a large data structure.
Moral: don't use amessage store when you really want a file store.
People complain the ZK's watches only give an update to latest state
sothey have to put messages into independent znodes and performance and
scalegoes to crap. Moral: don't use a synchronization system to store
messages.
Getting architectural basics close to write goes a long way.
[1] 
https://www.usenix.org/legacy/event/usenix10/tech/full_papers/Hunt.pdf
[2] https://ai.google/research/pubs/pub41684


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message