Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of ev@ooyala.com designates
 209.85.160.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOeJiJjzbOrBPvrQS0Rj=q39Q9AgfkqYc_qFQW5TuAi9-5hgcQ@mail.gmail.com>
References: 
 <CAOG_4QZWjqJstEJMeRi__6+=MOAmaepSUO60npH25et4OouaXQ@mail.gmail.com>
	<CAOeJiJjzbOrBPvrQS0Rj=q39Q9AgfkqYc_qFQW5TuAi9-5hgcQ@mail.gmail.com>
Date: Thu, 14 Jun 2012 13:39:16 -0700
Message-ID: 
 <CADWPM3gZEis=bcT7JUPBO0NKyyeiRroV5WGhodNE5QFF8WPASQ@mail.gmail.com>
Subject: Re: Consumer re-design proposal
From: Evan Chan <ev@ooyala.com>
To: kafka-users@incubator.apache.org
Cc: kafka-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=14dae934128977d9fd04c274b1c8

--14dae934128977d9fd04c274b1c8
Content-Type: text/plain; charset=ISO-8859-1

I would like to throw in a couple use cases:


   - Allow the new consumer to reset its offset to either the current
   largest or smallest.  This would be a great way to restart a process that
   has fallen behind.  The only way I know how to do this today, with the
   high-level consumer, is to delete the ZK nodes manually and restart the
   consumer.
   - Allow the consumer to reset its offset to some arbitrary value, and
   then write that offset into ZK.    Kind of like the first case, but would
   make rewinding/replays much easier.

Modularity (the ability to layer the ZK infrastructure on top of the simple
interface) would be great.

thanks,
Evan


On Tue, Jun 12, 2012 at 9:59 AM, Jay Kreps <jay.kreps@gmail.com> wrote:

> This is a great summary Neha. It would be good to get people's feedback on
> this since we don't want to keep breaking api and
> protocol compatibility here, so the hope is to really get it right this
> time now that we have really seen all the use cases and live with the
> output for a while. I think the consumer design is a pretty hard protocol
> and API design problem, so its fun to think about.
>
> If I were to summarize Neha's requirements list, I think there are three
> high-level goals:
>
>   1. Simplify the consumer protocol to enable ease of development of
>   consumer clients in other languages
>   2. Try to replace the "simple consumer" and "high level consumer" with a
>   single, general interface that has all the advantages of both.
>   3. Support a bunch of use cases that either we didn't think of, or that
>   weren't possible in the partitioning model of the pre-0.8 code base.
>
> -Jay
>
>
> On Mon, Jun 11, 2012 at 4:52 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Hi,
> >
> > Over the past few months, we've received quite a lot of feedback on the
> > consumer side features and design. Some of them are improvements to the
> > current consumer design and some are simply new feature/API requests. I
> > have attempted to write up the requirements that I've heard on this wiki
> -
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> >
> > This would involve some significant changes to the consumer APIs, so we
> > would like to collect feedback on the proposal from our community. Since
> > the list of changes is not small, we would like to understand if some
> > features are preferred over others, and more importantly, if some
> features
> > are not required at all.
> >
> > Since some part of this proposal is experimental and the consumer side
> > changes are non-trivial, we would like this initiative to not interfere
> > with the forthcoming replication release. However, it will be good to
> have
> > people from the community give this some thought and help out with the
> > JIRAs if interested. One way of managing this project could be creating a
> > separate branch from the kafka trunk and continue development on it. Once
> > it is ready and in good shape, we can think about cutting another release
> > (after 0.8) for the releasing the new consumer API. Do people have
> > preferences/concerns regarding creating a separate branch for this
> project
> > ?
> >
> > Please feel free to start a discussion on this JIRA -
> > https://issues.apache.org/jira/browse/KAFKA-364
> >
> > Thanks,
> >
> > Neha
> >
>


-- 
--
*Evan Chan*
Senior Software Engineer |
ev@ooyala.com | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

--14dae934128977d9fd04c274b1c8--