Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
Sender: ismaelj@gmail.com
In-Reply-To: <CAMQuQBbK3+gwGpZWmu429OG1B=8xgYF2Q94fHgfNWDUH_dpz1w@mail.gmail.com>
References: <CAD5tkZbUv_h+b5K+ngyO8cZ+V5j9iR0XhvFUf9WHQot_FuxrBw@mail.gmail.com>
 <786d35ca-d8c4-9781-d057-c46a5de2afc5@gmail.com> <CAD5tkZZUZ3YP6ZaUs=V9PW6mCwPfZbxehYs_GUnSbtRdksO3Dw@mail.gmail.com>
 <CAHwHRrVyaBcrxt6o9t9rbgTNf8jVYevTndhz3URPsMBWowo7xw@mail.gmail.com>
 <CAA9QSJKsbx3zHPiL1WNXxj_4wJ6WOK608wvty1ukAL0A6=S-fg@mail.gmail.com>
 <CAD5tkZa2TSpv5y-iUQy6CQ+u_rUPjJU95AKoLtqrvLY+TrtuhA@mail.gmail.com>
 <CABPX=-7sGL9CP3iCgX+=4kP1iLAwc_SmBs-af6Dj29XgYNAXeA@mail.gmail.com> <CAMQuQBbK3+gwGpZWmu429OG1B=8xgYF2Q94fHgfNWDUH_dpz1w@mail.gmail.com>
From: Ismael Juma <ismael@juma.me.uk>
Date: Mon, 20 Nov 2017 11:59:32 +0000
Message-ID: <CAD5tkZYZ11XYKDAsFwMRe3jRMXN8x+x2Wieyn3CcWLm12rky0g@mail.gmail.com>
Subject: Re: [DISCUSS] Kafka 2.0.0 in June 2018
To: dev <dev@kafka.apache.org>, Onur Karaman <okaraman@linkedin.com>
Content-Type: multipart/alternative; boundary="94eb2c142b362dcdd4055e68d477"
archived-at: Mon, 20 Nov 2017 12:00:28 -0000

--94eb2c142b362dcdd4055e68d477
Content-Type: text/plain; charset="UTF-8"

Thanks for the update Onur. Are you and the other committers and
contributors from LinkedIn planning to push this over the line?

Ismael

On Fri, Nov 10, 2017 at 9:53 PM, Onur Karaman <onurkaraman.apache@gmail.com>
wrote:

> Hey everyone. Regarding the status of KIP-125, just a heads up: I have an
> implementation of KIP-125 (KAFKA-4513) here:
> https://github.com/onurkaraman/kafka/commit/3b5448006ab70ba2b0b5e177853d19
> 1d0f777452
>
> The code might need to be rebased. The steps described in the KIP are a bit
> involved. Other than that, the implementation might have a bug with respect
> to converting arbitrary blacklist regexes to whitelist regexes since the
> new consumer only accepts whitelists.
>
> On Fri, Nov 10, 2017 at 11:36 AM, Jeff Widman <jeff@jeffwidman.com> wrote:
>
> > Re: migrating offsets for old Scala consumers.
> >
> > I work in the python world, so haven't directly used the old high level
> > consumer, but from what I understand the underlying problem remains the
> > migration of zookeeper offsets to the __consumer_offsets topic.
> >
> > We've used a slightly modified version of Grant Henke's script for
> > migrating offsets here: https://github.com/apache/kafka/pull/2615
> > It doesn't support rolling upgrades, but other than that it's great...
> I've
> > used it for multiple migrations, and very thankful for the time Grant put
> > into it.
> >
> > I don't know that it's worth pulling this into core, it might be, it
> might
> > not be. But it probably is worth documenting the procedure at least
> > somewhere.
> >
> > Personally, I suspect that those who absolutely need a rolling migration
> > and cannot handle a short period of downtime while doing a migration
> > probably have in-house experts on Kafka who are familiar with the issues
> > and willing to figure out a solution. The rest of the world can generally
> > handle a short maintenance window.
> >
> >
> >
> >
> > On Fri, Nov 10, 2017 at 10:46 AM, Ismael Juma <ismael@juma.me.uk> wrote:
> >
> > > Hi Gwen,
> > >
> > > A KIP has been proposed, but it is stalled:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-125%3A+
> > > ZookeeperConsumerConnector+to+KafkaConsumer+Migration+and+Rollback
> > >
> > > Unless the interested parties pick that up, we would drop support
> > without a
> > > rolling upgrade path. Users would be able to use the old consumers from
> > > 1.1.x for a long time. The old Scala clients don't support the message
> > > format introduced in 0.11.0, so the feature set is pretty much frozen
> and
> > > there's little benefit in upgrading. But there is a cost in keeping
> them
> > in
> > > the codebase.
> > >
> > > Ismael
> > >
> > > On Fri, Nov 10, 2017 at 6:02 PM, Gwen Shapira <gwen@confluent.io>
> wrote:
> > >
> > > > Last time we tried deprecating the Scala consumer, there were
> concerns
> > > > about a lack of upgrade path. There is no rolling upgrade, and
> > migrating
> > > > offsets is not trivial (and not documented).
> > > >
> > > > Did anything change in that regard? Or are we planning on dropping
> > > support
> > > > without an upgrade path?
> > > >
> > > >
> > > > On Fri, Nov 10, 2017 at 5:37 PM Guozhang Wang <wangguoz@gmail.com>
> > > wrote:
> > > >
> > > > > Thanks Ismael, the proposal looks good to me.
> > > > >
> > > > > A side note regarding: https://issues.apache.org/
> > > jira/browse/KAFKA-5637,
> > > > > could we resolve this ticket sooner than later to make clear about
> > the
> > > > code
> > > > > deprecation and support duration when moving from 1.0.x to 2.0.x?
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Fri, Nov 10, 2017 at 3:44 AM, Ismael Juma <ismael@juma.me.uk>
> > > wrote:
> > > > >
> > > > > > Features for 2.0.0 will be known after 1.1.0 is released in
> > February
> > > > > 2018.
> > > > > > We are still doing the usual time-based release process[1].
> > > > > >
> > > > > > I am raising this well ahead of time because of the potential
> > impact
> > > of
> > > > > > removing the old Scala clients (particularly the old high-level
> > > > consumer)
> > > > > > and dropping support for Java 7. Hopefully users can then plan
> > > > > accordingly.
> > > > > > We would do these changes in trunk soon after 1.1.0 is released
> > > (around
> > > > > > February).
> > > > > >
> > > > > > I think it makes sense to complete some of the work that was not
> > > ready
> > > > in
> > > > > > time for 1.0.0 (Controller improvements and JBOD are two that
> come
> > to
> > > > > mind)
> > > > > > in 1.1.0 (January 2018) and combined with the desire to give
> > advance
> > > > > > notice, June 2018 was the logical choice.
> > > > > >
> > > > > > There is no plan to support a particular release for longer. 1.x
> > > versus
> > > > > 2.x
> > > > > > is no different than 0.10.x versus 0.11.x from the perspective of
> > > > > > supporting older releases.
> > > > > >
> > > > > > [1] https://cwiki.apache.org/confluence/display/KAFKA/Time+
> > > > > > Based+Release+Plan
> > > > > >
> > > > > > On Fri, Nov 10, 2017 at 11:21 AM, Jaikiran Pai <
> > > > jai.forums2013@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Ismael,
> > > > > > >
> > > > > > > Are there any new features other than the language specific
> > changes
> > > > > that
> > > > > > > are being planned for 2.0.0? Also, when 2.x gets released, will
> > the
> > > > 1.x
> > > > > > > series see continued bug fixes and releases in the community or
> > is
> > > > the
> > > > > > plan
> > > > > > > to have one single main version that gets continuous updates
> and
> > > > > > releases?
> > > > > > >
> > > > > > > By the way, why June 2018? :)
> > > > > > >
> > > > > > > -Jaikiran
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 09/11/17 3:14 PM, Ismael Juma wrote:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >> I'm starting this discussion early because of the potential
> > > impact.
> > > > > > >>
> > > > > > >> Kafka 1.0.0 was just released and the focus was on achieving
> the
> > > > > > original
> > > > > > >> project vision in terms of features provided while maintaining
> > > > > > >> compatibility for the most part (i.e. we did not remove
> > deprecated
> > > > > > >> components like the Scala clients).
> > > > > > >>
> > > > > > >> This was the right decision, in my opinion, but it's time to
> > start
> > > > > > >> thinking
> > > > > > >> about 2.0.0, which is an opportunity for us to remove major
> > > > deprecated
> > > > > > >> components and to benefit from Java 8 language enhancements
> (so
> > > that
> > > > > we
> > > > > > >> can
> > > > > > >> move faster). So, I propose the following for Kafka 2.0.0:
> > > > > > >>
> > > > > > >> 1. It should be released in June 2018
> > > > > > >> 2. The Scala clients (Consumer, SimpleConsumer, Producer,
> > > > > SyncProducer)
> > > > > > >> will be removed
> > > > > > >> 3. Java 8 or higher will be required, i.e. support for Java 7
> > will
> > > > be
> > > > > > >> dropped.
> > > > > > >>
> > > > > > >> Thoughts?
> > > > > > >>
> > > > > > >> Ismael
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > *Jeff Widman*
> > jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
> > <><
> >
>

--94eb2c142b362dcdd4055e68d477--