Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of mumrah@gmail.com designates
 209.85.212.47 as permitted sender)
From: David Arthur <mumrah@gmail.com>
Mime-Version: 1.0 (Apple Message framework v1278)
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_8AF9BBA8-A5AB-4BC3-90F6-79308CAD0403"
Subject: Re: Kafka REST interface
Date: Mon, 10 Sep 2012 09:49:09 -0400
In-Reply-To: <E17EFDAC-EF9D-4000-8CAA-9CD297D70E1D@gmail.com>
To: kafka-dev@incubator.apache.org
References: <C3B23E4E-84D7-432B-9F5F-317CB936FDDA@gmail.com>
 <CAE_KnHLhAgQpYhLzpwaWoNckujO=-JbVcjxgiSaQEj-Lb6=iXA@mail.gmail.com>
 <892F7AA8-73FF-4FC3-9942-7BCA4F49F3C6@gmail.com>
 <CAOeJiJgpgV-OTHG5VXxGgkgPOQVS_n4Y+5vrQVxXyBf0oDvkwg@mail.gmail.com>
 <4256187394745870824@unknownmsgid>
 <E17EFDAC-EF9D-4000-8CAA-9CD297D70E1D@gmail.com>
Message-Id: <36333345-9B3C-45D3-B32B-19CB23C28A44@gmail.com>

--Apple-Mail=_8AF9BBA8-A5AB-4BC3-90F6-79308CAD0403
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1

Bump.=20

Anyone have feedback on this approach?

-David

On Aug 24, 2012, at 12:37 PM, David Arthur wrote:

> Here is an initial pass at a Kafka REST proxy (in Scala)
>=20
> =
https://github.com/mumrah/kafka/blob/rest/contrib/rest-proxy/src/main/scal=
a/RESTServer.scala
>=20
> The basic gist is:
> * Jetty for webserver
> * Messages are strings
> * GET /topic/group to get a message (timeout after 1s)
> * POST /topic, the request body is the message
> * One consumer thread per topic+group
>=20
> Be wary, many things are hard coded at this point (port numbers, etc). =
Obviously, this will need to change. Also, I haven't the slightest idea =
how to setup/use sbt properly, so I just checked in the libs.
>=20
> Feedback is welcome in this thread or on Github.  Be gentle please, =
this is my first go at Scala
>=20
> -David
>=20
> On Aug 12, 2012, at 10:39 AM, Taylor Gautier wrote:
>=20
>> Jay I agree with you 100%.
>>=20
>> At Tagged we have implemented a proxy for various internal reasons (
>> primarily to act as a high performance relay from PHP to Kafka). It's
>> implemented in Node.js (JavaScript)
>>=20
>> Currently it services UDP packets encoded in binary but it could
>> easily be modified to accept http also since Node support for http is
>> pretty simple.
>>=20
>> If others are interested in maintaining something like this we could
>> consider adding this to the public domain along side the already
>> existing Node.js client implementation.
>>=20
>>=20
>>=20
>> On Aug 10, 2012, at 3:51 PM, Jay Kreps <jay.kreps@gmail.com> wrote:
>>=20
>>> My personal preference would be to have only a single protocol in =
kafka
>>> core. I have been down the multiple protocol route and my experience =
was
>>> that it adds a lot of burden for each change that needs to be made =
and a
>>> lot of complexity to abstract over the different protocols. =46rom =
the point
>>> of view of a user they are generally a bit agnostic as to how bytes =
are
>>> sent back and forth provided it is reliable and easily implementable =
in any
>>> language. Generally they care more about the quality of the client =
in their
>>> language of choice.
>>>=20
>>> My belief is that the main benefit of REST is ease of implementing a
>>> client. But currently the biggest barrier is really the use of zk =
and
>>> fairly thick consumer design. So I think the current thinking is =
that we
>>> should focus on thinning that out and removing the client-side zk
>>> dependency. I actually don't think TCP is a huge burden if the =
protocol is
>>> simple, and there are actually some advantages (for example the =
consumer
>>> needs to consume from multiple servers so select/poll/epoll is =
natural but
>>> this is not always available from HTTP client libraries).
>>>=20
>>> Basically this is an area where I think it is best to pick one way =
and
>>> really make it really bullet proof rather than providing lots of =
options.
>>> In some sense each option tends to increase the complexity of =
testing
>>> (since now there are many combinations to try) and also of =
implementation
>>> (since now a lot things that were concrete now need to be abstracted =
away).
>>>=20
>>> So from this perspective I would prefer a standalone proxy that =
could
>>> evolve independently rather than retro-fitting the current socket =
server to
>>> handle other protocols. There will be some overhead for the extra =
hop, but
>>> then there is some overhead for HTTP itself.
>>>=20
>>> This is just my personal opinion, it would be great to hear what =
other
>>> think.
>>>=20
>>> -Jay
>>>=20
>>> On Mon, Aug 6, 2012 at 5:39 AM, David Arthur <mumrah@gmail.com> =
wrote:
>>>=20
>>>> I'd be happy to collaborate on this, though it's been a while since =
I've
>>>> used PHP.
>>>>=20
>>>> =46rom what it looks like, what you have is a true proxy that runs =
outside
>>>> of Kafka and translates some REST routes into Kafka client calls. =
This
>>>> sounds more in line with what the project page describes. What I =
have
>>>> proposed is more like a translation layer between some REST routes =
and
>>>> FetchRequests. In this case the client is responsible for managing =
offsets.
>>>> Using the consumer groups and ZooKeeper would be another nice way =
of
>>>> consuming messages (which is probably more like what you have).
>>>>=20
>>>> Any maintainers have feedback on this?
>>>>=20
>>>> On Aug 3, 2012, at 4:13 PM, Jonathan Creasy wrote:
>>>>=20
>>>>> I have an internal one working and was hoping to have it open =
sourced in
>>>>> the next week. The one at Box is based on the CodeIgniter =
framework, we
>>>>> have about 45 RESTful interfaces built on this framework so I just =
put
>>>>> together another one for Kafka.
>>>>>=20
>>>>>=20
>>>>> Here are my notes, these were pre-dev so may be a little different =
than
>>>>> what we ended up with.
>>>>>=20
>>>>> =
https://cwiki.apache.org/confluence/display/KAFKA/Restful+API+Proposal
>>>>>=20
>>>>> I will read yours later this afternoon, we should work together.
>>>>>=20
>>>>> -Jonathan
>>>>>=20
>>>>>=20
>>>>> On Fri, Aug 3, 2012 at 7:41 AM, David Arthur <mumrah@gmail.com> =
wrote:
>>>>>=20
>>>>>> I'd like to tackle this project (assuming it hasn't been started =
yet).
>>>>>>=20
>>>>>> I wrote up some initial thoughts here: =
https://gist.github.com/3248179
>>>>>>=20
>>>>>> TLDR;  use Range header for specifying offsets, simple URIs like
>>>>>> /kafka/topics/[topic]/[partition], use for a simple transport of =
bytes
>>>>>> and/or represent the messages as some media type (text, json, =
xml)
>>>>>>=20
>>>>>> Feedback is most welcome (in the Gist or in this thread).
>>>>>>=20
>>>>>> Cheers!
>>>>>>=20
>>>>>> -David
>>>>=20
>>>>=20
>=20


--Apple-Mail=_8AF9BBA8-A5AB-4BC3-90F6-79308CAD0403--