Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of jay.kreps@gmail.com designates
 209.85.213.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <892F7AA8-73FF-4FC3-9942-7BCA4F49F3C6@gmail.com>
References: <C3B23E4E-84D7-432B-9F5F-317CB936FDDA@gmail.com>
	<CAE_KnHLhAgQpYhLzpwaWoNckujO=-JbVcjxgiSaQEj-Lb6=iXA@mail.gmail.com>
	<892F7AA8-73FF-4FC3-9942-7BCA4F49F3C6@gmail.com>
Date: Fri, 10 Aug 2012 12:50:42 -0700
Message-ID: 
 <CAOeJiJgpgV-OTHG5VXxGgkgPOQVS_n4Y+5vrQVxXyBf0oDvkwg@mail.gmail.com>
Subject: Re: Kafka REST interface
From: Jay Kreps <jay.kreps@gmail.com>
To: kafka-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=e89a8ff256e4c7e7f104c6eea886

--e89a8ff256e4c7e7f104c6eea886
Content-Type: text/plain; charset=ISO-8859-1

My personal preference would be to have only a single protocol in kafka
core. I have been down the multiple protocol route and my experience was
that it adds a lot of burden for each change that needs to be made and a
lot of complexity to abstract over the different protocols. From the point
of view of a user they are generally a bit agnostic as to how bytes are
sent back and forth provided it is reliable and easily implementable in any
language. Generally they care more about the quality of the client in their
language of choice.

My belief is that the main benefit of REST is ease of implementing a
client. But currently the biggest barrier is really the use of zk and
fairly thick consumer design. So I think the current thinking is that we
should focus on thinning that out and removing the client-side zk
dependency. I actually don't think TCP is a huge burden if the protocol is
simple, and there are actually some advantages (for example the consumer
needs to consume from multiple servers so select/poll/epoll is natural but
this is not always available from HTTP client libraries).

Basically this is an area where I think it is best to pick one way and
really make it really bullet proof rather than providing lots of options.
In some sense each option tends to increase the complexity of testing
(since now there are many combinations to try) and also of implementation
(since now a lot things that were concrete now need to be abstracted away).

So from this perspective I would prefer a standalone proxy that could
evolve independently rather than retro-fitting the current socket server to
handle other protocols. There will be some overhead for the extra hop, but
then there is some overhead for HTTP itself.

This is just my personal opinion, it would be great to hear what other
think.

-Jay

On Mon, Aug 6, 2012 at 5:39 AM, David Arthur <mumrah@gmail.com> wrote:

> I'd be happy to collaborate on this, though it's been a while since I've
> used PHP.
>
> From what it looks like, what you have is a true proxy that runs outside
> of Kafka and translates some REST routes into Kafka client calls. This
> sounds more in line with what the project page describes. What I have
> proposed is more like a translation layer between some REST routes and
> FetchRequests. In this case the client is responsible for managing offsets.
> Using the consumer groups and ZooKeeper would be another nice way of
> consuming messages (which is probably more like what you have).
>
> Any maintainers have feedback on this?
>
> On Aug 3, 2012, at 4:13 PM, Jonathan Creasy wrote:
>
> > I have an internal one working and was hoping to have it open sourced in
> > the next week. The one at Box is based on the CodeIgniter framework, we
> > have about 45 RESTful interfaces built on this framework so I just put
> > together another one for Kafka.
> >
> >
> > Here are my notes, these were pre-dev so may be a little different than
> > what we ended up with.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Restful+API+Proposal
> >
> > I will read yours later this afternoon, we should work together.
> >
> > -Jonathan
> >
> >
> > On Fri, Aug 3, 2012 at 7:41 AM, David Arthur <mumrah@gmail.com> wrote:
> >
> >> I'd like to tackle this project (assuming it hasn't been started yet).
> >>
> >> I wrote up some initial thoughts here: https://gist.github.com/3248179
> >>
> >> TLDR;  use Range header for specifying offsets, simple URIs like
> >> /kafka/topics/[topic]/[partition], use for a simple transport of bytes
> >> and/or represent the messages as some media type (text, json, xml)
> >>
> >> Feedback is most welcome (in the Gist or in this thread).
> >>
> >> Cheers!
> >>
> >> -David
>
>

--e89a8ff256e4c7e7f104c6eea886--