kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix GV <fe...@mate1inc.com>
Subject Re: Kafka 0.8 producer within Play Framework?
Date Tue, 05 Feb 2013 18:24:16 GMT
Hmm interesting...... combining Play and Kafka is a cool combination :) ...

Being completely stateless, Play nodes can go up and down without any
issue. If you're relying on an async producer, however, then you kind of
lose this property of Play, because unpublished messages could be lost in
the event of a node failure (which means the nodes become stateful, within
a small time bound). If the production is sync and non-batched, then the
Play statelessness is maximized, but Kafka's throughput is going to be
severely limited with such settings. Of course, you could have a sync batch
producer, which would be properly stateless and allow Kafka to have a
potentially high throughput, but then Play's throughput will be severely
limited because calls to Kafka will be blocking, and Play is meant to have
few threads that process a high throughput of (shot lived) non-blocking
operations, not a high amount of threads that can each do blocking
operations however they want.

A re-usable async batch producer is what should give you the best possible
throughput for both Play and Kafka, at the expense of risking a bit of data
loss in the event of a failure... In the end, this is pretty much the same
compromise we always make I guess...

--
Felix


On Tue, Feb 5, 2013 at 10:13 AM, charlie w <spambait132@gmail.com> wrote:

> Yes probably, but ATM, I've just been trying to get it to work at all.
>  Fixing issues like that is on the way...
>
> On Tue, Feb 5, 2013 at 7:38 AM, S Ahmed <sahmed1020@gmail.com> wrote:
> > Shouldn't your producer be at the controller scope?  Instantiating it
> every
> > time is probably nto the correct pattern.  You probably want to use a
> asych
> > producer also right?
> >
> >
> >
> >
> > On Mon, Feb 4, 2013 at 7:12 PM, charlie w <spambait132@gmail.com> wrote:
> >
> >> It seems the issue is related to Scala versions.  I grabbed from
> >> https://github.com/kevinwright/kafka
> >> and built with
> >> sbt "++2.10.0 package"
> >>
> >> I'm able to have my Play app produce messages for Kafka now.
> >>
> >>
> >> On Fri, Feb 1, 2013 at 5:32 PM, charlie w <spambait132@gmail.com>
> wrote:
> >> > Is it possible to have a Kafka 0.8 producer inside a Play Framework
> >> > controller?  (I am new to both Kafka and Play.)
> >> >
> >> > I have managed to get Java code cribbed from the Kafka examples
> >> > (below) to compile and execute within Play, but constructing the
> >> > ProducerConfig object never returns.  I am not finding anything in any
> >> > Play log.
> >> >
> >> > I've also tried this using the Scala console producer as a model and
> >> > had the same result
> >> >
> >> > The Kafka console producer and consumer samples are working for me.
> >> >
> >> > Thanks
> >> > Charlie
> >> >
> >> > code:
> >> >
> >> >
> >> > package controllers;
> >> >
> >> > import java.util.Properties;
> >> >
> >> > import kafka.producer.ProducerConfig;
> >> > import kafka.javaapi.producer.Producer;
> >> > import kafka.producer.KeyedMessage;
> >> >
> >> > public class Hello
> >> > {
> >> >    public static void hello( String v )
> >> >    {
> >> >       Properties props = new Properties();
> >> >       props.put("zk.connect", "127.0.0.1:2181");
> >> >       props.put("serializer.class", "kafka.serializer.StringEncoder");
> >> >       props.put("broker.list", "localhost:9092" );
> >> >
> >> >       play.Logger.info("create config");
> >> >       ProducerConfig config = new ProducerConfig(props);
> >> >
> >> >       play.Logger.info("create producer");
> >> >       Producer<Integer, String> producer = new Producer<Integer,
> >> > String>(config);
> >> >
> >> >       play.Logger.info("yay!");
> >> >
> >> >       KeyedMessage msg = new KeyedMessage<Integer,
> >> > String>("test-topic", "Greetings!");
> >> >
> >> >       producer.send( msg );
> >> >    }
> >> > }
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message