Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of mumrah@gmail.com designates
 209.85.213.47 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1278)
Subject: Re: Kafka REST interface
From: David Arthur <mumrah@gmail.com>
In-Reply-To: 
 <CAOeJiJgpgV-OTHG5VXxGgkgPOQVS_n4Y+5vrQVxXyBf0oDvkwg@mail.gmail.com>
Date: Fri, 10 Aug 2012 23:15:57 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <E4AC360E-0050-40FE-8726-A561CCA220C6@gmail.com>
References: <C3B23E4E-84D7-432B-9F5F-317CB936FDDA@gmail.com>
 <CAE_KnHLhAgQpYhLzpwaWoNckujO=-JbVcjxgiSaQEj-Lb6=iXA@mail.gmail.com>
 <892F7AA8-73FF-4FC3-9942-7BCA4F49F3C6@gmail.com>
 <CAOeJiJgpgV-OTHG5VXxGgkgPOQVS_n4Y+5vrQVxXyBf0oDvkwg@mail.gmail.com>
To: kafka-dev@incubator.apache.org

You make a good point about TCP not being the problem. But rather the =
complex consumer logic and ZK dependency are barriers to entry for new =
clients. I'm starting to like the idea of a standalone proxy that uses =
an existing client (one with batteries included) to simply things for =
HTTP clients. If this is going into Kafka, I think we should stick with =
Scala/Java for implementation.=20

Are there any preferences for what HTTP server is used? Any particular =
aversion to Jetty?=20

-David

On Aug 10, 2012, at 3:50 PM, Jay Kreps wrote:

> My personal preference would be to have only a single protocol in =
kafka
> core. I have been down the multiple protocol route and my experience =
was
> that it adds a lot of burden for each change that needs to be made and =
a
> lot of complexity to abstract over the different protocols. =46rom the =
point
> of view of a user they are generally a bit agnostic as to how bytes =
are
> sent back and forth provided it is reliable and easily implementable =
in any
> language. Generally they care more about the quality of the client in =
their
> language of choice.
>=20
> My belief is that the main benefit of REST is ease of implementing a
> client. But currently the biggest barrier is really the use of zk and
> fairly thick consumer design. So I think the current thinking is that =
we
> should focus on thinning that out and removing the client-side zk
> dependency. I actually don't think TCP is a huge burden if the =
protocol is
> simple, and there are actually some advantages (for example the =
consumer
> needs to consume from multiple servers so select/poll/epoll is natural =
but
> this is not always available from HTTP client libraries).
>=20
> Basically this is an area where I think it is best to pick one way and
> really make it really bullet proof rather than providing lots of =
options.
> In some sense each option tends to increase the complexity of testing
> (since now there are many combinations to try) and also of =
implementation
> (since now a lot things that were concrete now need to be abstracted =
away).
>=20
> So from this perspective I would prefer a standalone proxy that could
> evolve independently rather than retro-fitting the current socket =
server to
> handle other protocols. There will be some overhead for the extra hop, =
but
> then there is some overhead for HTTP itself.
>=20
> This is just my personal opinion, it would be great to hear what other
> think.
>=20
> -Jay
>=20
> On Mon, Aug 6, 2012 at 5:39 AM, David Arthur <mumrah@gmail.com> wrote:
>=20
>> I'd be happy to collaborate on this, though it's been a while since =
I've
>> used PHP.
>>=20
>> =46rom what it looks like, what you have is a true proxy that runs =
outside
>> of Kafka and translates some REST routes into Kafka client calls. =
This
>> sounds more in line with what the project page describes. What I have
>> proposed is more like a translation layer between some REST routes =
and
>> FetchRequests. In this case the client is responsible for managing =
offsets.
>> Using the consumer groups and ZooKeeper would be another nice way of
>> consuming messages (which is probably more like what you have).
>>=20
>> Any maintainers have feedback on this?
>>=20
>> On Aug 3, 2012, at 4:13 PM, Jonathan Creasy wrote:
>>=20
>>> I have an internal one working and was hoping to have it open =
sourced in
>>> the next week. The one at Box is based on the CodeIgniter framework, =
we
>>> have about 45 RESTful interfaces built on this framework so I just =
put
>>> together another one for Kafka.
>>>=20
>>>=20
>>> Here are my notes, these were pre-dev so may be a little different =
than
>>> what we ended up with.
>>>=20
>>> =
https://cwiki.apache.org/confluence/display/KAFKA/Restful+API+Proposal
>>>=20
>>> I will read yours later this afternoon, we should work together.
>>>=20
>>> -Jonathan
>>>=20
>>>=20
>>> On Fri, Aug 3, 2012 at 7:41 AM, David Arthur <mumrah@gmail.com> =
wrote:
>>>=20
>>>> I'd like to tackle this project (assuming it hasn't been started =
yet).
>>>>=20
>>>> I wrote up some initial thoughts here: =
https://gist.github.com/3248179
>>>>=20
>>>> TLDR;  use Range header for specifying offsets, simple URIs like
>>>> /kafka/topics/[topic]/[partition], use for a simple transport of =
bytes
>>>> and/or represent the messages as some media type (text, json, xml)
>>>>=20
>>>> Feedback is most welcome (in the Gist or in this thread).
>>>>=20
>>>> Cheers!
>>>>=20
>>>> -David
>>=20
>>=20