Return-Path: 
 <kafka-dev-return-3165-apmail-incubator-kafka-dev-archive=incubator.apache.org@incubator.apache.org>
X-Original-To: apmail-incubator-kafka-dev-archive@minotaur.apache.org
Delivered-To: apmail-incubator-kafka-dev-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D372D915B
	for <apmail-incubator-kafka-dev-archive@minotaur.apache.org>;
 Tue, 12 Jun 2012 16:59:36 +0000 (UTC)
Received: (qmail 55667 invoked by uid 500); 12 Jun 2012 16:59:36 -0000
Delivered-To: apmail-incubator-kafka-dev-archive@incubator.apache.org
Received: (qmail 55631 invoked by uid 500); 12 Jun 2012 16:59:36 -0000
Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:kafka-dev-help@incubator.apache.org>
List-Unsubscribe: <mailto:kafka-dev-unsubscribe@incubator.apache.org>
List-Post: <mailto:kafka-dev@incubator.apache.org>
List-Id: <kafka-dev.incubator.apache.org>
Reply-To: kafka-dev@incubator.apache.org
Delivered-To: mailing list kafka-dev@incubator.apache.org
Received: (qmail 55598 invoked by uid 99); 12 Jun 2012 16:59:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jun 2012 16:59:36 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jay.kreps@gmail.com designates
 209.85.161.175 as permitted sender)
Received: from [209.85.161.175] (HELO mail-gg0-f175.google.com)
 (209.85.161.175)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jun 2012 16:59:28 +0000
Received: by ggnp4 with SMTP id p4so3409852ggn.6
        for <multiple recipients>; Tue, 12 Jun 2012 09:59:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        bh=+zxpVYXj5+OQBo7cEEfB+ZP4RKbO3I0OzLd5ak1RjLo=;
        b=06wKOXJHso3ibkL2W4OgvyA8CEITcEgd4/retd8S76q+RX3Eye99w0nkoPCyil0eyp
         1OdvARP+z99iACtTEds1ZE410Zxqv0H4teMYmTUE6RfMHeRyh5wy3tXPJ5rJp7bR0uuY
         DzvWcE+L8AN2uTXriy6bxlNg4KdA/4CQC10qTI3AgqSMhIa8RYIP1G93nNMhKXZYsNiG
         umxjMZBEO9XqZwyFZSy+t9K302BM+YD8N2FYrsnoFRJvEAMAlA00R0jrjMz7ODF4ivfa
         NCKqiCyqCUw6qZ3FSxkf+zT0WwnCt1a2lQy5KqtCf4o0ypiy5M8ETjQEeh719xun62mT
         1P0g==
MIME-Version: 1.0
Received: by 10.50.76.137 with SMTP id k9mr8962278igw.25.1339520347250; Tue,
 12 Jun 2012 09:59:07 -0700 (PDT)
Received: by 10.231.22.27 with HTTP; Tue, 12 Jun 2012 09:59:07 -0700 (PDT)
In-Reply-To: 
 <CAOG_4QZWjqJstEJMeRi__6+=MOAmaepSUO60npH25et4OouaXQ@mail.gmail.com>
References: 
 <CAOG_4QZWjqJstEJMeRi__6+=MOAmaepSUO60npH25et4OouaXQ@mail.gmail.com>
Date: Tue, 12 Jun 2012 09:59:07 -0700
Message-ID: 
 <CAOeJiJjzbOrBPvrQS0Rj=q39Q9AgfkqYc_qFQW5TuAi9-5hgcQ@mail.gmail.com>
Subject: Re: Consumer re-design proposal
From: Jay Kreps <jay.kreps@gmail.com>
To: kafka-users@incubator.apache.org, kafka-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=e89a8f23433b779ff104c24962bb

--e89a8f23433b779ff104c24962bb
Content-Type: text/plain; charset=ISO-8859-1

This is a great summary Neha. It would be good to get people's feedback on
this since we don't want to keep breaking api and
protocol compatibility here, so the hope is to really get it right this
time now that we have really seen all the use cases and live with the
output for a while. I think the consumer design is a pretty hard protocol
and API design problem, so its fun to think about.

If I were to summarize Neha's requirements list, I think there are three
high-level goals:

   1. Simplify the consumer protocol to enable ease of development of
   consumer clients in other languages
   2. Try to replace the "simple consumer" and "high level consumer" with a
   single, general interface that has all the advantages of both.
   3. Support a bunch of use cases that either we didn't think of, or that
   weren't possible in the partitioning model of the pre-0.8 code base.

-Jay


On Mon, Jun 11, 2012 at 4:52 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Hi,
>
> Over the past few months, we've received quite a lot of feedback on the
> consumer side features and design. Some of them are improvements to the
> current consumer design and some are simply new feature/API requests. I
> have attempted to write up the requirements that I've heard on this wiki -
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
>
> This would involve some significant changes to the consumer APIs, so we
> would like to collect feedback on the proposal from our community. Since
> the list of changes is not small, we would like to understand if some
> features are preferred over others, and more importantly, if some features
> are not required at all.
>
> Since some part of this proposal is experimental and the consumer side
> changes are non-trivial, we would like this initiative to not interfere
> with the forthcoming replication release. However, it will be good to have
> people from the community give this some thought and help out with the
> JIRAs if interested. One way of managing this project could be creating a
> separate branch from the kafka trunk and continue development on it. Once
> it is ready and in good shape, we can think about cutting another release
> (after 0.8) for the releasing the new consumer API. Do people have
> preferences/concerns regarding creating a separate branch for this project
> ?
>
> Please feel free to start a discussion on this JIRA -
> https://issues.apache.org/jira/browse/KAFKA-364
>
> Thanks,
>
> Neha
>

--e89a8f23433b779ff104c24962bb--