Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of mjuarez@gmail.com designates
 209.85.213.47 as permitted sender)
Subject: Re: Consumer re-design proposal
Mime-Version: 1.0 (Apple Message framework v1278)
Content-Type: text/plain; charset=iso-8859-1
From: Marcos Juarez <mjuarez@gmail.com>
In-Reply-To: 
 <CADWPM3gZEis=bcT7JUPBO0NKyyeiRroV5WGhodNE5QFF8WPASQ@mail.gmail.com>
Date: Thu, 14 Jun 2012 15:45:07 -0600
Cc: kafka-dev@incubator.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <09378B01-AEF3-4E4F-8861-AE7622F49339@gmail.com>
References: 
 <CAOG_4QZWjqJstEJMeRi__6+=MOAmaepSUO60npH25et4OouaXQ@mail.gmail.com>
 <CAOeJiJjzbOrBPvrQS0Rj=q39Q9AgfkqYc_qFQW5TuAi9-5hgcQ@mail.gmail.com>
 <CADWPM3gZEis=bcT7JUPBO0NKyyeiRroV5WGhodNE5QFF8WPASQ@mail.gmail.com>
To: kafka-users@incubator.apache.org

Throwing a +1 on "Allow the consumer to reset its offset to some =
arbitrary value, and then write that offset into ZK".

We're currently running into a scenario where we would like to have 100% =
reliability, and we're losing a few messages when a connection is =
broken, but there were still a few messages in the OS TCP buffers. So, =
we're planning on shifting the ZK offset by a few seconds "back in time" =
if we detect a broker has gone down, to make sure all the messages will =
be actually delivered to the end consumer when that broker comes back =
up, even if there's a small amount of overlapping messages.

Thanks,

Marcos


On Jun 14, 2012, at 2:39 PM, Evan Chan wrote:

> I would like to throw in a couple use cases:
>=20
>=20
>   - Allow the new consumer to reset its offset to either the current
>   largest or smallest.  This would be a great way to restart a process =
that
>   has fallen behind.  The only way I know how to do this today, with =
the
>   high-level consumer, is to delete the ZK nodes manually and restart =
the
>   consumer.
>   - Allow the consumer to reset its offset to some arbitrary value, =
and
>   then write that offset into ZK.    Kind of like the first case, but =
would
>   make rewinding/replays much easier.
>=20
> Modularity (the ability to layer the ZK infrastructure on top of the =
simple
> interface) would be great.
>=20
> thanks,
> Evan
>=20
>=20
> On Tue, Jun 12, 2012 at 9:59 AM, Jay Kreps <jay.kreps@gmail.com> =
wrote:
>=20
>> This is a great summary Neha. It would be good to get people's =
feedback on
>> this since we don't want to keep breaking api and
>> protocol compatibility here, so the hope is to really get it right =
this
>> time now that we have really seen all the use cases and live with the
>> output for a while. I think the consumer design is a pretty hard =
protocol
>> and API design problem, so its fun to think about.
>>=20
>> If I were to summarize Neha's requirements list, I think there are =
three
>> high-level goals:
>>=20
>>  1. Simplify the consumer protocol to enable ease of development of
>>  consumer clients in other languages
>>  2. Try to replace the "simple consumer" and "high level consumer" =
with a
>>  single, general interface that has all the advantages of both.
>>  3. Support a bunch of use cases that either we didn't think of, or =
that
>>  weren't possible in the partitioning model of the pre-0.8 code base.
>>=20
>> -Jay
>>=20
>>=20
>> On Mon, Jun 11, 2012 at 4:52 PM, Neha Narkhede =
<neha.narkhede@gmail.com
>>> wrote:
>>=20
>>> Hi,
>>>=20
>>> Over the past few months, we've received quite a lot of feedback on =
the
>>> consumer side features and design. Some of them are improvements to =
the
>>> current consumer design and some are simply new feature/API =
requests. I
>>> have attempted to write up the requirements that I've heard on this =
wiki
>> -
>>>=20
>> =
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Desig=
n
>>>=20
>>> This would involve some significant changes to the consumer APIs, so =
we
>>> would like to collect feedback on the proposal from our community. =
Since
>>> the list of changes is not small, we would like to understand if =
some
>>> features are preferred over others, and more importantly, if some
>> features
>>> are not required at all.
>>>=20
>>> Since some part of this proposal is experimental and the consumer =
side
>>> changes are non-trivial, we would like this initiative to not =
interfere
>>> with the forthcoming replication release. However, it will be good =
to
>> have
>>> people from the community give this some thought and help out with =
the
>>> JIRAs if interested. One way of managing this project could be =
creating a
>>> separate branch from the kafka trunk and continue development on it. =
Once
>>> it is ready and in good shape, we can think about cutting another =
release
>>> (after 0.8) for the releasing the new consumer API. Do people have
>>> preferences/concerns regarding creating a separate branch for this
>> project
>>> ?
>>>=20
>>> Please feel free to start a discussion on this JIRA -
>>> https://issues.apache.org/jira/browse/KAFKA-364
>>>=20
>>> Thanks,
>>>=20
>>> Neha
>>>=20
>>=20
>=20
>=20
>=20
> --=20
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>