kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin McCabe" <cmcc...@apache.org>
Subject Re: [DISCUSS] Kafka 3.0
Date Mon, 11 May 2020 20:39:35 GMT
On Mon, May 11, 2020, at 01:19, David Jacot wrote:
> Hi all,
> 
> First, I agree with what has been discussed. Having 3.x as the bridge
> releases and entirely removing ZK in 4.0 makes total sense.
> 
> Second, what would you think about removing the auto topics creation
> in 3.0? It is not recommended to use it anymore and that could simplify
> a bit our path towards removing ZK. We could deprecate it in 2.6 already
> and remove it in 3.0. If you guys generally agree, I could draft a KIP and
> kick off the discussion.
>

I like the idea of removing automatic topic creation, but it seems like it would be a big
step, given that it's currently on by default.

What about if we changed it to be off by default and deprecated in 3.0?  Then, people who
hadn't yet changed their workflows could turn it on in 3.0, and we could remove it entirely
in 4.0.

best,
Colin

> 
> Best,
> David
> 
> On Sun, May 10, 2020 at 8:02 PM Michael K. Edwards <m.k.edwards@gmail.com>
> wrote:
> 
> > Yes, I've read the KIP.  But all it really says to me is "we have never
> > gotten around to using ZooKeeper properly."  To the extent that any of the
> > distributed-state-maintenance problems discussed in "Metadata as an Event
> > Log" can be solved — and some of them intrinsically can't, because CAP
> > theorem — most of them are already implemented very effectively in Curator
> > recipes.  (For instance, Curator's Tree Cache
> > https://curator.apache.org/curator-recipes/tree-cache.html is a good fit
> > to
> > some of the state-maintenance needs.)
> >
> > Kafka does have some usage patterns that don't map neatly onto existing
> > Curator recipes.  For instance, neither LeaderSelector nor LeaderLatch
> > implements leader preference in the way that the existing Kafka partition
> > leadership election procedure does.  But why not handle that by improving
> > and extending Curator?  That way, other Curator users benefit, and we get
> > additional highly experienced reviewers' eyes on the distributed
> > algorithms, which are very very tricky to get right.
> >
> >
> > On Sun, May 10, 2020 at 10:47 AM Ron Dagostino <rndgstn@gmail.com> wrote:
> >
> > > Hi Michael.  This is discussed in the KIP.
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation
> > >
> > > Ron
> > >
> > > > On May 10, 2020, at 1:35 PM, Michael K. Edwards <m.k.edwards@gmail.com
> > >
> > > wrote:
> > > >
> > > > What is the actual goal of removing the ZooKeeper dependency?  In my
> > > > experience, if ZooKeeper is properly provisioned and deployed, it's
> > > largely
> > > > trouble-free.  (You do need to know how to use observers properly.)
> > > There
> > > > are some subtleties about timeouts and leadership changes, but they're
> > > > pretty small stuff.  Why go to all the trouble of building a new
> > > > distributed-consensus system that's going to have catastrophic bugs for
> > > > years to come?  It seems like such an act of hubris to me, as well as
a
> > > > massive waste of engineering effort.  What is there to be gained?
> > > >
> > > >> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax <mjsax@apache.org>
> > > wrote:
> > > >>
> > > >> Sure, we can compile a list for Kafka Streams. But the KIP would be
> > for
> > > >> 3.0, so I don't think it's urgent to do it now?
> > > >>
> > > >>
> > > >> -Matthias
> > > >>
> > > >>> On 5/8/20 3:47 PM, Colin McCabe wrote:
> > > >>> Thanks, Guozhang-- sounds like a good plan.
> > > >>>
> > > >>> I think it would be good to have a list of deprecated streams
APIs
> > that
> > > >> we want to remove in 3.0.  Maybe it's easiest to do that as its own
> > KIP?
> > > >>>
> > > >>> For MirrorMaker 1, we should have a KIP to deprecate its use in
2.6
> > if
> > > >> we want to remove it in 3.0.  I don't have a good sense of how
> > > practical it
> > > >> is to deprecate this now, so I will defer to others here.  But the
KIP
> > > >> freeze for 2.6 is coming soon, so if we want to make the case, now
is
> > > the
> > > >> time.
> > > >>>
> > > >>> best,
> > > >>> Colin
> > > >>>
> > > >>>
> > > >>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote:
> > > >>>> Hey folks,
> > > >>>>
> > > >>>> Sorry for stating that the bridge release would not break
any
> > > >> compatibility
> > > >>>> before, which is incorrect and confused many people.
> > > >>>>
> > > >>>> I think one way to think about the versioning is that:
> > > >>>>
> > > >>>> 0) In a 2.x version moving ahead we would deprecate the ZK-dependent
> > > >> tools
> > > >>>> such as --zookeeper flags from various scripts (KIP-555)
> > > >>>>
> > > >>>> 1) In 3.0 we would at least make one incompatible change for
example
> > > to
> > > >>>> remove the deprecated ZK flags.
> > > >>>>
> > > >>>> 2) In a future major version (e.g. 4.0) we would drop ZK entirely,
> > > >>>> including usages such as security credentials / broker registration
> > /
> > > >> etc
> > > >>>> which are via ZK today as well.
> > > >>>>
> > > >>>> Then for the bridge release(s), it can be any or all of 3.x.
> > > >>>>
> > > >>>>
> > > >>>> For 1), I'd love to add a few more incompatibility changes
in 3.0
> > from
> > > >>>> Kafka Streams: we evolve Streams public APIs by deprecating
and then
> > > >> remove
> > > >>>> in major releases, and since 2.0 we've accumulated quite a
few
> > > >> deprecated
> > > >>>> APIs, and I can compile a list of KIPs that contain those
if people
> > > are
> > > >>>> interested.
> > > >>>>
> > > >>>>
> > > >>>> Guozhang
> > > >>>>
> > > >>>>
> > > >>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe <cmccabe@apache.org>
> > > wrote:
> > > >>>>>
> > > >>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote:
> > > >>>>>>> In fact, we know that the bridge release will
involve at least
> > one
> > > >>>>>>> incompatible change.  We will need to drop support
for the
> > > >> --zookeeper
> > > >>>>>>> flags in the command-line tools.
> > > >>>>>>
> > > >>>>>> If the bridge release(s) and the subsequent post-ZK
release are
> > > _both_
> > > >>>>>> breaking changes, I think we only have one option:
the 3.x line
> > are
> > > >> the
> > > >>>>>> bridge release(s), and ZK is removed in 4.0, as suggested
by
> > Andrew
> > > >>>>>> Schofield.
> > > >>>>>>
> > > >>>>>> Specifically:
> > > >>>>>> - in order to _remove_ (not merely deprecate) the
--zookeeper
> > args,
> > > we
> > > >>>>> will
> > > >>>>>> need a major release.
> > > >>>>>> - in oder to drop support for ZK entirely (e.g. break
a bunch of
> > > >> external
> > > >>>>>> tooling like Cruise Control), we will need a major
release.
> > > >>>>>>
> > > >>>>>> I count two major releases.
> > > >>>>>
> > > >>>>> Hi Ryanne,
> > > >>>>>
> > > >>>>> I agree that dropping ZK completely will need a new major
release
> > > after
> > > >>>>> 3.0.  I think that's OK and in keeping with how we've
handled
> > > >> deprecation
> > > >>>>> and removal in the past.  It's important for users to
have a smooth
> > > >> upgrade
> > > >>>>> path.
> > > >>>>>
> > > >>>>> best,
> > > >>>>> Colin
> > > >>>>>
> > > >>>>>>
> > > >>>>>> Ryanne
> > > >>>>>>
> > > >>>>>> -
> > > >>>>>>
> > > >>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe <cmccabe@apache.org>
> > > >> wrote:
> > > >>>>>>
> > > >>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan
wrote:
> > > >>>>>>>> Hey Colin, I think we should wait until after
KIP-500's "bridge
> > > >>>>>>>> release" so there is a clean break from Zookeeper
after 3.0. The
> > > >>>>>>>> bridge release by definition is an attempt
to not break
> > anything,
> > > so
> > > >>>>>>>> it theoretically doesn't warrant a major release.
> > > >>>>>>>
> > > >>>>>>> Hi Ryanne,
> > > >>>>>>>
> > > >>>>>>> I think it's important to clarify this a little
bit.  The bridge
> > > >>>>> release
> > > >>>>>>> (really, releases, plural) allow you to upgrade
from a cluster
> > that
> > > >> is
> > > >>>>>>> using ZooKeeper to one that is not using ZooKeeper.
 But, that
> > > >> doesn't
> > > >>>>>>> imply that the bridge release itself doesn't break
anything.
> > > >> Upgrading
> > > >>>>>>> to the bridge release itself might involve some
minor
> > > >> incompatibility.
> > > >>>>>>>
> > > >>>>>>> Kafka does occasionally have incompatible changes.
 In those
> > cases,
> > > >> we
> > > >>>>>>> bump the major version number.  One example is
that when we went
> > > from
> > > >>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for
JDK7.  This is an
> > > >>>>>>> incompatible change.
> > > >>>>>>>
> > > >>>>>>> In fact, we know that the bridge release will
involve at least
> > one
> > > >>>>>>> incompatible change.  We will need to drop support
for the
> > > >> --zookeeper
> > > >>>>>>> flags in the command-line tools.
> > > >>>>>>>
> > > >>>>>>> We've been preparing for this change for a long
time.  People
> > have
> > > >>>>> spent
> > > >>>>>>> a lot of effort designing new APIs that can be
used instead of
> > the
> > > >> old
> > > >>>>>>> zookeeper-based code that some of the command-line
tools used.
> > We
> > > >> have
> > > >>>>>>> also deprecated the old ZK-based flags.  But at
the end of the
> > day,
> > > >> it
> > > >>>>>>> is still an incompatible change.  So it's unfortunately
not
> > > possible
> > > >>>>> for
> > > >>>>>>> the
> > > >>>>>>> bridge release to be a 2.x release.
> > > >>>>>>>
> > > >>>>>>>> If that's not the case (i.e. if a single "bridge
release" turns
> > > out
> > > >>>>> to
> > > >>>>>>>> be impractical), we should consider forking
3.0 while
> > maintaining
> > > a
> > > >>>>>>>> line of Zookeeper-dependent Kafka in 2.x.
That way 3.x can
> > evolve
> > > >>>>>>>> dramatically without breaking the 2.x line.
In particular,
> > > anything
> > > >>>>>>>> related to removing Zookeeper could land in
pre-3.0 while every
> > > >> other
> > > >>>>>>>> feature targets 2.6.
> > > >>>>>>>
> > > >>>>>>> Just to be super clear about this, what we want
to do here is
> > > support
> > > >>>>>>> operating in __either__ KIP-500 mode and legacy
mode for a while.
> > > So
> > > >>>>> the
> > > >>>>>>> same branch will have support for both the old
way and the new
> > way
> > > of
> > > >>>>>>> managing metadata.
> > > >>>>>>>
> > > >>>>>>> This will allow us to get an "alpha" version of
the KIP-500 mode
> > > out
> > > >>>>> early
> > > >>>>>>> for people to experiment with.  It also greatly
reduces the
> > number
> > > of
> > > >>>>> Kafka
> > > >>>>>>> releases we have to make, and the amount of backporting
we have
> > to
> > > >> do.
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> If you are proposing 2.6 should be the "bridge
release", I think
> > > >> this
> > > >>>>>>>> is premature given Kafka's time-based release
schedule. If the
> > > >> bridge
> > > >>>>>>>> features happen to be merged before 2.6's
feature freeze, then
> > > sure
> > > >>>>> --
> > > >>>>>>>> let's make that the bridge release in retrospect.
And if we get
> > > all
> > > >>>>>>>> the post-Zookeeper features merged before
2.7, I'm onboard with
> > > >>>>> naming
> > > >>>>>>>> it "3.0" instead.
> > > >>>>>>>>
> > > >>>>>>>> That said, we should aim to remove legacy
MirrorMaker before 3.0
> > > as
> > > >>>>>>>> well. I'm happy to drive that additional breaking
change. Maybe
> > > 2.6
> > > >>>>>>>> can be the "bridge" for MM2 as well.
> > > >>>>>>>
> > > >>>>>>> I don't have a strong opinion either way about
this, but if we
> > want
> > > >> to
> > > >>>>>>> remove the original MirrorMaker, we have to deprecate
it first,
> > > >>>>> right?  Are
> > > >>>>>>> we ready to do that?
> > > >>>>>>>
> > > >>>>>>> best,
> > > >>>>>>> Colin
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Ryanne
> > > >>>>>>>>
> > > >>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe
<cmccabe@apache.org>
> > > >>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi all,
> > > >>>>>>>>>
> > > >>>>>>>>> We've had a few proposals recently for
incompatible changes.
> > One
> > > >>>>> of
> > > >>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags
from the
> > > Administrative
> > > >>>>>>>>> Tools.  The other is Boyang's KIP-590:
Redirect ZK Mutation
> > > >>>>>>>>> Protocols to the Controller.  I think
it's time to start
> > thinking
> > > >>>>>>>>> about Kafka 3.0. Specifically, I think
we should move to 3.0
> > > after
> > > >>>>>>>>> the 2.6 release.
> > > >>>>>>>>>
> > > >>>>>>>>> From the perspective of KIP-500, in Kafka
3.x we'd like to make
> > > >>>>>>>>> running in a ZooKeeper-less mode possible
(but not yet the
> > > >>>>> default.)
> > > >>>>>>>>> This is the motivation behind KIP-590
and KIP-604, as well as
> > > some
> > > >>>>>>>>> of the other KIPs we've done recently.
 Since it will take some
> > > >>>>> time
> > > >>>>>>>>> to stabilize the new ZooKeeper-free Kafka
code, we will hide it
> > > >>>>>>>>> behind an option initially. (We'll have
a KIP describing this
> > all
> > > >>>>> in
> > > >>>>>>>>> detail soon.)
> > > >>>>>>>>>
> > > >>>>>>>>> What does everyone think about having
Kafka 3.0 come up next
> > > after
> > > >>>>>>>>> 2.6? Are there any other things we should
change in the 2.6 ->
> > > 3.0
> > > >>>>>>>>> transition?
> > > >>>>>>>>>
> > > >>>>>>>>> best, Colin
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> -- Guozhang
> > > >>>>
> > > >>
> > > >>
> > >
> >
>

Mime
View raw message