kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Davison <ben.davi...@7digital.com>
Subject Re: [DISCUSS] KIP-80: Kafka REST Server
Date Sun, 02 Oct 2016 21:45:39 GMT
+ 1 to rest client (don't mind if it's the current confluent version or
something else)

We are a multi language company and the quality of the other clients that
are not Java are really hit and miss. A rest endpoint a user could just
pump messages into or subscribe to would be amazing.

On Sun, Oct 2, 2016 at 9:23 PM, Harsha Chintalapani <kafka@harsha.io> wrote:

> Neha & Jay,
>                  We did look at the open source alternatives. Our concern
> is what's the patch acceptance and adding features/ bug-fixes to the
> existing project under a Github (although it's licensed under Apache 2.0).
> It would be great if that project made available under Apache and driven by
> the community.  Adding to the above, not all Kafka users are interested in
> using the Java client API, they would like to have simple REST API where
> they can code against using any language. I do believe this adds value to
> Apache Kafka in itself.
>
> "For 1, I don't think there is value in giving in to the NIH syndrome and
> reinventing the wheel. What I'm looking for is a detailed comparison of the
> gaps and why those can't be improved in the REST proxy that already exists
> and is actively maintained."
>
> We are not looking at this as  NIH. What we are worried about is a project
> that's not maintained in a community. So the process of accepting patches
> and priorities is not clear, and it's not developed in Apache community.
> Not only that, existing REST API project doesn't support new client API and
> hence there is no security support either.
> We don't know the timeline when that's made available. We would like to add
> admin functionality into the REST API. So the Roadmap of that project is
> not driven by Apache.
>
>
> "This doesn't materially have an impact on expanding the usability of
>    Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
>    Kafka users, and maybe 10% of those are the ones who will use the REST
>    proxy. The remaining 50% are non-java client users (C, python, go, node
>    etc)."
>
> REST API is most often asked feature in my interactions with Kafka users.
> In an organization, There will be independent teams who will write their
>  Kafka clients using different language libraries available today, and
> there is no way to standardize this. Instead of supporting several
> different client libraries users will be interested in using a REST API
> server. The need for a REST API will only increase as more and more users
> start using Kafka.
>
> "More surface area means more work to keep things consistent. Failure
>    to do that has, in fact, hurt the user experience."
> Having myriad Kafka client GitHub projects that support different languages
> hurts the user experience and pushes burden to maintain these libraries.
> REST API is a simple code base that uses existing java client libraries to
> make life easier for the users.
>
> Thanks,
> Harsha
>
> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <neha@confluent.io> wrote:
>
> > Manikumar,
> >
> > Thanks for sharing the proposal. I think there are 2 parts to this
> > discussion -
> >
> > 1. Should we rewrite a REST proxy given that there is a feature-complete,
> > open-source and actively maintained REST proxy in the community?
> > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> maintain
> > the high-quality experience that Kafka users have today?
> >
> > For 1, I don't think there is value in giving in to the NIH syndrome and
> > reinventing the wheel. What I'm looking for is a detailed comparison of
> the
> > gaps and why those can't be improved in the REST proxy that already
> exists
> > and is actively maintained. For example, we depend on zkClient and have
> > found as well as fixed several bugs by working closely with the people
> who
> > maintain zkClient. This should be possible for REST proxy as well, right?
> >
> > For 2, I'd like us to review our history of expanding the surface area to
> > add more clients in the past. Here is a summary -
> >
> >    - This doesn't materially have an impact on expanding the usability of
> >    Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> > all
> >    Kafka users, and maybe 10% of those are the ones who will use the REST
> >    proxy. The remaining 50% are non-java client users (C, python, go,
> node
> >    etc).
> >    - People are a lot more excited about promising to contribute while
> >    adding the surface area but not on an ongoing basis down the line.
> >    - More surface area means more work to keep things consistent. Failure
> >    to do that has, in fact, hurt the user experience.
> >    - More surface area hurts agility. We want to do a few things really
> >    well as well as be agile to be able to build on our core competency.
> >
> >
> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <manikumar.reddy@gmail.com>
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Thanks for your reply.
> > >
> > > I agree that we can not add all the clients/tools available in
> ecosystem
> > > page to Kafka repo itself. But we feel REST Interface is different from
> > > other clients/tools. Since any language that can work with HTTP can
> > > easily integrate with this interface, Having an "official"  REST
> > > interface helps user community. This also helps us to integrate well
> > > with external management and provisioning tools.  Apache Kafka release
> > > with Java clients + REST interface is sufficient for most of the user
> > > deployments/requirements. This helps users to deal with less number
> > > of distributions/builds.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > >
> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <jay@confluent.io> wrote:
> > >
> > > > Hey guys,
> > > >
> > > > There's already a REST interface maintained as a separate
> project--it's
> > > > open source and apache licensed and actively maintained (
> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
> that?
> > > You
> > > > mentioned that there was some compatibility concern, but
> compatibility
> > > has
> > > > to do with the consumer protocol guarantees not the repo the code is
> > in,
> > > > right? Not sure that concern makes sense.
> > > >
> > > > We could argue for adding pretty much anything and everything in the
> > > > ecosystem page in Kafka itself but I'm not sure that would make the
> > > project
> > > > more agile.
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> manikumar.reddy@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Kafka Devs,
> > > > >
> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > > >
> > > > > There are already open-source alternatives are available.  But we
> > would
> > > > > like to add REST server that
> > > > > many users ask for under Apache Kafka repo. Many data Infra tools
> > comes
> > > > up
> > > > > with Rest Interface.
> > > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > > messages
> > > > > and admin interface for
> > > > > integrating with external management and provisioning tools.This
> will
> > > > also
> > > > > allow the maintenance of
> > > > > REST server and adding new features makes it easy because apache
> > > > community.
> > > > >
> > > > > The KIP wiki is the following:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 80%3A+Kafka+Rest+Server
> > > > >
> > > > > Your comments and feedback are welcome.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> > --
> > Thanks,
> > Neha
> >
>

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message