kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: [DISCUSS] Streams-Broker compatibility regression in 2.2.1 release
Date Mon, 16 Sep 2019 23:22:15 GMT
+1 to the proposal. Let's also highlight this in the release notes for
2.2.2 and 2.3.0 please.

Ismael

On Wed, Sep 11, 2019 at 10:23 AM Matthias J. Sax <matthias@confluent.io>
wrote:

> Hi,
>
> recently a user reported an issue upgrading a Kafka Streams application
> from 2.2.0 to 2.2.1 (cf
> https://mail-archives.apache.org/mod_mbox/kafka-users/201908.mbox/
> <CAG8RrxkDAhCrGQ1%3DpBpX-Jcxagk_xkmRk4MBQeZYAP6ZGHoK_w%40mail.gmail.com>)
>
> After some investigation, we identified
> https://issues.apache.org/jira/browse/KAFKA-7895 to be the root cause of
> the problem.
>
> The fix for KAFKA-7895 is using message headers and thus requires broker
> version 0.11.0 (or newer) and message format 0.11 (or newer). Hence,
> while a Kafka Streams application version 2.2.0 is compatible to older
> brokers (0.10.1 and 0.10.2) and only requires message format 0.10, the
> backward compatibility was broken accidentally in 2.2.1.
>
> The fix is also contained in 2.3.0 release and cherry-picked to 2.1
> branch (2.1.2 is not released yet, and thus 2.1 users are not affected
> as this point).
>
> Note: only users that use `suppress()` operator in their program are
> affected.
>
> We should not break streams-broker backward compatibility in bug-fix
> releases at all and avoid for minor releases. However, it seems
> necessary to have the fix in 2.3.0 though -- otherwise, `suppress()` is
> effectively useless and it does not seem to be a good idea to fix the
> bug only in the next major release. Hence, trading-off some backward
> compatibility in a minor release seems to be acceptable for this case,
> considering that 0.11.0 was release 2 years ago.
>
> For 2.2.1, it is more challenging to decide how to move forward, because
> we should not have broken streams-broker compatibility but 2.2.1 is
> already released and we can only react after the fact.
>
>   From my point of view, the best way is to keep the fix and update the
> release notes and documentation accordingly. The main reason for my
> suggestions is that we would expect a majority of users to be on 0.11.0
> brokers already and the fix will work for them -- reverting the fix in
> 2.2.2 seems to be worse for all those users on newer broker versions. We
> also know that `suppress()` is a highly demanded feature and a lot of
> people waited for a fix.
>
>   The expected minority of users that are on 0.10.1 / 0.10.2 brokers, or
> newer brokers but still on message format 0.10 would either need to stay
> on Kafka Streams 2.2.0 or upgrade their brokers/message format
> accordingly. However, upgrading brokers/message format is de-facto
> already required for 2.2.1 and thus keeping the fix would not make the
> situation worse.
>
> For 2.1, I would suggest to revert the fix to make sure we don't break
> streams-broker compatibility for 2.1.x users. If those users need the
> fix for `suppress()` they need to upgrade to 2.2.1/2.3.0 or newer and
> make sure their brokers are on 0.11.0 with message format 0.11, too.
>
>
> TL;DR; the proposal is:
>
> (1) revert the fix for KAFKA-7895 in 2.1 branch
> (2) keep the fix for KAFKA-7895 in 2.2.1 and 2.3.0
>
> Impact:
>
>  - Kafka Streams 2.1.x and 2.2.0 applications are backward compatible
> back to 0.10.1 brokers, requiring message format 0.10
>  - Kafka Streams 2.2.2 / 2.3.0 application (or newer) are backward
> compatible back to 0.11.0 brokers, requiring message format 0.11
>
>
> Please let us know what you think about this proposal.
>
>
> -Matthias
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message