kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Arthur <mum...@gmail.com>
Subject Re: Kafka REST interface
Date Fri, 10 Aug 2012 16:54:42 GMT
In regards to embedding an HTTP server in Kafka to handle REST requests, how hard would it
be to modify/extend the existing SocketServer? Seems like lots of good work went into the
networking stuff in Kafka, it would make sense to try and leverage that. I could imagine KafkaServer
(optionally) starting up an HttpSocketServer with HttpRequestHandlers (similar to the existing
SocketServer/RequestHandlers). Is this feasible/sensible?

Assuming Kafka handles the socket layer, perhaps something like Apache HTTP Components (http://hc.apache.org)
could be used for parsing the HTTP messages? I'd like to stay away from higher level web frameworks
for something simple like this.


On Aug 6, 2012, at 2:19 PM, Jonathan Creasy wrote:

> That is correct, a Consume REST request would be handed off to a Consumer class which
communicates with ZooKeeper and Kafka. The proxy keeps track of clients, there are a few routes
to manipulate the offsets if you wanted to force a reset, seek to the end, or re-retrieve
some data from a topic. 
> 
> If others agree, I would be happy to collaborate on integrating a REST interface to Kakfa
directly, more as your have proposed, rather than a proxy. The proxy was low-hanging fruit
within Box because we already had a framework for cranking out a REST API quite easily and
a set of Kafka+ZK classes. It was simply stitching the two together.
> 
> -Jonathan
> 
> On Mon, Aug 6, 2012 at 5:39 AM, David Arthur <mumrah@gmail.com> wrote:
> I'd be happy to collaborate on this, though it's been a while since I've used PHP.
> 
> From what it looks like, what you have is a true proxy that runs outside of Kafka and
translates some REST routes into Kafka client calls. This sounds more in line with what the
project page describes. What I have proposed is more like a translation layer between some
REST routes and FetchRequests. In this case the client is responsible for managing offsets.
Using the consumer groups and ZooKeeper would be another nice way of consuming messages (which
is probably more like what you have).
> 
> Any maintainers have feedback on this?
> 
> On Aug 3, 2012, at 4:13 PM, Jonathan Creasy wrote:
> 
> > I have an internal one working and was hoping to have it open sourced in
> > the next week. The one at Box is based on the CodeIgniter framework, we
> > have about 45 RESTful interfaces built on this framework so I just put
> > together another one for Kafka.
> >
> >
> > Here are my notes, these were pre-dev so may be a little different than
> > what we ended up with.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Restful+API+Proposal
> >
> > I will read yours later this afternoon, we should work together.
> >
> > -Jonathan
> >
> >
> > On Fri, Aug 3, 2012 at 7:41 AM, David Arthur <mumrah@gmail.com> wrote:
> >
> >> I'd like to tackle this project (assuming it hasn't been started yet).
> >>
> >> I wrote up some initial thoughts here: https://gist.github.com/3248179
> >>
> >> TLDR;  use Range header for specifying offsets, simple URIs like
> >> /kafka/topics/[topic]/[partition], use for a simple transport of bytes
> >> and/or represent the messages as some media type (text, json, xml)
> >>
> >> Feedback is most welcome (in the Gist or in this thread).
> >>
> >> Cheers!
> >>
> >> -David
> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message