nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <christophe.mon...@post.ch>
Subject RE: kafka-clients version
Date Wed, 17 Jun 2020 06:35:17 GMT
Hi Joe

According to https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix "newer
versions of the Java clients and other clients that support KIP-35 such as librdkafka are
able to fall back to older request types or throw appropriate errors if functionality is not
available. This flexibility is a big win for users, who now have much more flexibility in
adopting new client versions without requiring broker upgrades first."

That's why I suggested to always pack the latest client version.

If in your experience both major and minor version matters, then NiFi should allow to precisely
select them.

At the moment we're stuck with client 2.0 while our brokers are being migrated from 2.4 to
2.5, so we're affected by issues like https://issues.apache.org/jira/browse/KAFKA-9815 and
the blocker is the NiFi distribution.

As a workaround we could replace kafka-clients.jar in nifi-kafka-bundle.nar, what do you think?

Chris

-----Original Message-----
From: Joe Witt <joe.witt@gmail.com> 
Sent: Friday, 12. June 2020 15:50
To: dev@nifi.apache.org
Subject: Re: kafka-clients version

Chris

In my experience with the clients the answer is a definite no.  An older client will generally
work (with important caveats at times) with a newer broker.  A newer client will generally
not work with an older broker.  At the very least both major and minor version of the client
matters so users can appropriately map to their broker.

We just need to put out a 2.5 one now

thanks

On Fri, Jun 12, 2020 at 6:46 AM <christophe.monnet@post.ch> wrote:

> Hi
>
> Please see the discussion below.
> Would it make sense to always pack the latest minor version of 
> kafka-clients-jar in nifi-kafka-bundle?
>
> The processors could be named “2.x” (or simply “2” really) and the 
> specific minor version can change in NiFi releases.
>
> Regards
> Chris
>
> From: Bryan Bende <bbende@gmail.com>
> Sent: Friday, 12. June 2020 15:33
> To: users@nifi.apache.org
> Subject: Re: kafka-clients version
>
> Gotcha. Yes, we can obviously change the version of the client lib, 
> but it's misleading because they were named "2_0" because they use the "2_0"
> client, and now that won't be true anymore, but if others agree with 
> this then maybe it's fine.
>
> Maybe more of a question for the dev list.
>
> On Fri, Jun 12, 2020 at 9:28 AM <christophe.monnet@post.ch<mailto:
> christophe.monnet@post.ch>> wrote:
> Yes: https://issues.apache.org/jira/browse/KAFKA-9815
>
> > we'd have to be sure the latest 2.x client also works well with all
> previous 2.x versions back to 2.0
>
> I think kafka-clients should make sure of that.
>
> The processors could still be named 2_0, it’s only the client jar that 
> should be the latest 2.x version.
> Specifically
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka
> -bundle/pom.xml#L31
>
> Regards
> Chris
>
> From: Bryan Bende <bbende@gmail.com<mailto:bbende@gmail.com>>
> Sent: Friday, 12. June 2020 15:18
> To: users@nifi.apache.org<mailto:users@nifi.apache.org>
> Subject: Re: kafka-clients version
>
> Usually we make a new version of the processors when something changes 
> that causes a previous version of the client to not work well with the 
> newer broker (i.e. 0.9 client with 0.10 broker, in the older versions).
>
> The latest processors are named with "2_0" so not sure if it makes 
> sense to upgrade the client lib and then the name is no longer 
> accurate, plus we'd have to be sure the latest 2.x client also works 
> well with all previous 2.x versions back to 2.0.
>
> The other option is to make a new copy of the processor like "2_5" or 
> whatever the latest version is, but I think we only want to do this if 
> it's really necessary. AFAIK the 2.0 client works with the latest 2.x brokers.
>
> Is there any specific issue you are looking to solve by using the 
> latest client?
>
>
> On Fri, Jun 12, 2020 at 8:20 AM <christophe.monnet@post.ch<mailto:
> christophe.monnet@post.ch>> wrote:
> Hi
>
> Any reason why nifi-kafka-bundle is still bundled with 
> kafka-clients-2.0.0.jar and not a newer version?
>
> Regards
> Chris
>
Mime
View raw message