kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Molnár Bálint <molnarcsi...@gmail.com>
Subject Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper
Date Wed, 04 Apr 2018 08:08:21 GMT
Hi,
Thanks again for the feedback.

Is there already ongoing work for having an own consensus implementation
within Kafka?
If that work haven't started yet, we think there is value in having an
interim solution, that allows the use of another consensus system besides
Zookeeper.

We ask the community to take a look at the Etcd implementation proposal  we
created and provide feedback on that.
This helps to asses rather this approach is viable at all.

We are open to collaborate on integrating our proposed Etcd implementation
into any integration test system, to certify that all use cases works as
expected.

Balint

2018-03-30 22:21 GMT+02:00 Gwen Shapira <gwen@confluent.io>:

> Hi,
>
> I had an offline discussion with Ismael and wanted to summarize the
> comments and questions he raised so we are all on the same page.
>
> The core issue is that this change adds a new public API. Since we already
> know that the goal for the next 1-2 years is to get rid of ZK completely.
> Do we want to go to the effort of adding (and discussing and reviewing) a
> new public API, knowing that it will be completely removed in a year? And
> since building and testing a plugin also involves effort, will anyone do it
> for something that is going to be temporary by design?
>
> Ismael, correct me if this isn't a fair representation of your concerns.
>
> Gwen
>
>
>
> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <gwen@confluent.io> wrote:
>
> > Few other concerns that were raised in the previous discussion were
> around
> > the challenges both to maintainers and users in making this API pluggable
> > and how does making the interface pluggable aligns with future goals for
> > the project. At the time this was difficult to discuss because there
> wasn't
> > a concrete proposal. I want to discuss these points in the context of
> this
> > specific proposal:
> >
> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple
> > implementations to cover.
> >     In this case: At the time, the Kafka project didn't have much
> > experience with pluggable APIs and components, so the concerns were very
> > valid. Right now Kafka has multiple pluggable components - Connectors,
> > converters, transformations, authentication protocols, authorization
> > database, coordination protocol, serializers, etc. I think that as a
> > community we gotten better at testing the interface, testing the very few
> > implementations that are included in Apache Kafka itself and allowing the
> > community to innovate and validate outside of the Kafka project. I don't
> > recall major issues either from lack of testing or from usability
> > perspective.
> >
> > 2. Problem: Users don't want to choose a consensus implementation, they
> > just don't want ZK.
> >     In this case: I agree that users don't actually want to spend time
> > choosing consensus implementation and a simpler deployment model would
> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK
> > implementation, 99% of the users will choose to use that (a vast majority
> > uses our less-than-amazing authorization plugin), and the few that really
> > need something else for whatever reason, will be able to get what they
> > need. As Jake said, we need to face the fact that development trajectory
> of
> > ZK isn't amazing at the moment, that it is lacking features our users
> need
> > (SSL) and it will be good to allow the user community to explore
> > alternatives.
> >
> > 3. Problem: Why got to the effort of refactoring if we know we want to
> get
> > rid of ZK.
> >     In this case: This change isn't huge, it doesn't rewrite large
> > portions of Kafka and it does not make the future direction any more
> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
> > solution, applying it on Kafka with this KIP isn't any more challenging
> > than applying it to Kafka without this KIP. It is a step in an orthogonal
> > direction, but not opposite direction. I think that letting the perfect
> > become the enemy of the good is a repeated failure mode in this
> community.
> > Can we discuss whether this proposal is good even if there is a more
> > complex proposal that may be better? As long as they don't conflict?
> >
> > Gwen
> >
> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint <molnarcsirke@gmail.com>
> > wrote:
> >
> >> Thanks, for the feedback.
> >>
> >> Developing an internal consensus service inside Kafka would require a
> team
> >> dedicated to this task.
> >> We second what Flavio said in
> >> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f
> >> 7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org%3E
> >> that
> >> getting an implementation which really works and is maintainable is
> >> difficult.
> >> We think that Kafka being able to use another widely used consensus
> system
> >> beside Zookeeper its a safer and workable solution.
> >> It will be easier for users to use a consensus system with Kafka that
> they
> >> are already familiar with.
> >>
> >>
> >> The implementation found here:
> >> https://github.com/banzaicloud/apache-kafka-on-k8s/tree/
> >> kafka-on-etcd/core/src/main/scala/kafka/etcd
> >> is a first version of enabling Etcd in Kafka.
> >> This implementation hooked in Etcd with a slight change in the existing
> >> interfaces. While this implementation works its far from being complete.
> >> Ideally existing interfaces should be reworked to abstract out the used
> >> consensus system.
> >> We opened this KIP to start a discussion and the community to have a
> look
> >> at the initial implementation and receive feedback if this initiative is
> >> viable.
> >>
> >> Balint
> >>
> >> 2018-03-29 11:23 GMT+02:00 Jakub Scholz <jakub@scholz.cz>:
> >>
> >> > I can understand the concerns about the plugability of Zookeeper/Etcd.
> >> It
> >> > would not be good for Kafka community if it splits into several groups
> >> > using different implementations.
> >> >
> >> > On the other hand, Zookeeper development seems to be a bit stalled. So
> >> > maybe there should be at least a discussion whether it makes sense to
> >> > replace Zookeeper with something like Etcd or not.
> >> >
> >> > JAkub
> >> >
> >> > On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint <
> molnarcsirke@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> >> > >
> >> > > Here is the link to the KIP:
> >> > >
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> >> > >
> >> > > Looking forward to the discussion.
> >> > >
> >> > > Thanks,
> >> > > Balint
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > <http://www.confluent.io/blog>
> >
> >
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message