kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Chan ...@ooyala.com>
Subject Re: Non-blocking Kafka stream iterators
Date Tue, 22 Jan 2013 17:37:40 GMT
Jay,

For the consumer:
- Separation of the consumer logic from the main logic
- Making it easier to build the consumer for different versions of Scala
(say 2.10)
- Make it easier to read from any offset you want, while being able to keep
partition management features
- Better support for Akka and other non-blocking / event-based frameworks
(instead of a timeout, implement true hasNext functionality, for example)

thanks,
Evan


On Mon, Jan 21, 2013 at 9:27 AM, Jay Kreps <jay.kreps@gmail.com> wrote:

> It's worth mentioning that we are interested in exploring potential
> generalizations of the producer and consumer API, but as a practical matter
> most of the committers are working on getting a stable 0.8 release out the
> door. So an improved consumer and producer api would be a 0.9 feature.
>
> If you have a concrete thing you are trying to do now that is awkward it
> would be great to hear about the use case.
>
> Possible goals of improving the apis and client impls would include the
> following:
>
> Producer:
> 1. Include the offset in the information returned to the producer
> 2. Pipeline producer requests to improve throughput for synchronous
> production
>
> Consumer
> 1. Simplify api while supporting various advanced use cases like
> multi-stream consumption
> 2. Make partition assignment optional and server-side (this is currently
> the difference between the zk consumer and the simple consumer)
> 3. Make offset management optional
> 4. Remove threading from the consumer
> 5. Simplify consumer memory management
>
> -Jay
>
>
>
>
> On Mon, Jan 21, 2013 at 8:05 AM, Jun Rao <junrao@gmail.com> wrote:
>
> > No, but you can implement it in your application.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Jan 20, 2013 at 11:02 PM, navneet sharma <
> > navneetsharma0505@gmail.com> wrote:
> >
> > > Is there any property to make consumer work for lets say only 10 mins
> (ie
> > > some kind of timer)
> > > So, i want to close the consumer after 10 mins reading from broker..
> > >
> > > Thanks,
> > > Navneet Sharma
> > >
> > >
> > > On Mon, Jan 21, 2013 at 11:48 AM, Ryan LeCompte <lecompte@gmail.com>
> > > wrote:
> > >
> > > > Perfect. Thanks Jun!
> > > >
> > > > Ryan
> > > >
> > > >
> > > > On Sun, Jan 20, 2013 at 10:16 PM, Jun Rao <junrao@gmail.com> wrote:
> > > >
> > > > > Yes, see consumer.timeout.ms in
> > > > http://kafka.apache.org/configuration.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Sun, Jan 20, 2013 at 7:14 PM, Ryan LeCompte <lecompte@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > Does Kafka still only support blocking stream iterstors? It
would
> > be
> > > > > great
> > > > > > to pass a timeout or have a poll() operation for fetching items.
> > > Right
> > > > > now
> > > > > > I'm always blocking in this call:
> > > > > > for (m <- stream) ...
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Ryan
> > > > >
> > > >
> > >
> >
>



-- 
--
*Evan Chan*
Senior Software Engineer |
ev@ooyala.com | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message