kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsha <ka...@harsha.io>
Subject Re: [VOTE] KIP-433: Block old clients on brokers
Date Wed, 24 Apr 2019 14:34:16 GMT
Hi Gwen & Ismael,
                   Do you have any feedback on with the proposed approach, min.api.version
allowing users to specify versions for every request.

Thanks,
Harsha

On Fri, Apr 19, 2019, at 10:24 AM, Harsha wrote:
> Thanks Ying for updating the KIP. 
> Hi Ismael,
>              Given min.api.version allows admin/users to specifiy 
> min.version for each request this should address your concerns right?
> 
> Thanks,
> Harsha
> 
> On Wed, Apr 17, 2019, at 2:29 PM, Ying Zheng wrote:
> > I have updated the config description in the KIP, made the example more
> > clear
> > 
> > The proposed change allows setting different min versions for different
> > APIs, and the ApiVersionRequest change is already in the KIP.
> > 
> > On Fri, Apr 12, 2019 at 8:22 PM Harsha <kafka@harsha.io> wrote:
> > 
> > > Hi Ismael,
> > >             I meant to say blocking clients based on their API version
> > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48
> > > But If I understand what you are saying, since each client release can
> > > support different versions for each of fetch, produce, offset commit etc..
> > > and it's harder to block just based on single min.api.version setting
> > > across different clients.
> > > The idea I had in my mind was to do this via ApiVersionRequest, when a
> > > client makes api request to broker in response we return min and max
> > > version supported for each Api. When min.api.version enabled on broker, it
> > > returns the maxVersion it supports for each of the requests in that release
> > > as min versions to the clients.
> > >
> > > Example:
> > > Kafka 1.1.1 broker and min.api.verson set to
> > > https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79
> > > (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for
> > > example produce request
> > >
> > > https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112
> > > Instead of returning all of the supported versions it will return
> > > PRODUCE_REQUEST_V5 as the only supported version.
> > >
> > > Irrespective of the above approach I understand your point still stands
> > > which is sarama might not choose to implement all the higher version
> > > protocols for Kafka 1.1 release and they might introduce higher version of
> > > produce request in a subsequent minor release and it will be harder for
> > > users to figure out which release of sarama client they can use.
> > >
> > >
> > > Ying, if you have a different apporach which might address this issue
> > > please add.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote:
> > > > Hi Harsha,
> > > >
> > > > There is no such thing as 1.1 protocol. I encourage you to describe an
> > > > example config that achieves what you are suggesting here. It's pretty
> > > > complicated because the versions are per API and each client evolves
> > > > independently.
> > > >
> > > > Ismael
> > > >
> > > > On Sat, Apr 13, 2019 at 4:09 AM Harsha <kafka@harsha.io> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > "Relying on min.version seems like a pretty clunky way to achieve
the
> > > above
> > > > > > list. The challenge is that it's pretty difficult to do it in
a way
> > > that
> > > > > > works for clients across languages. They each add support for
new
> > > > > protocol
> > > > > > versions independently (it could even happen in a bug fix release).
> > > So,
> > > > > if
> > > > > > you tried to block Sarama in #2, you may block Java clients
too."
> > > > >
> > > > > That's the intended effect, right?  if you as the admin/operator
> > > > > configures the broker to have min.api.version to be 1.1
> > > > > it should block java , sarama clients etc.. which are below the 1.1
> > > > > protocol.  As mentioned this is not just related to log.format upgrade
> > > > > problem but in general a forcing cause to get the users to upgrade
> > > their
> > > > > client version in a multi-tenant environment.
> > > > >
> > > > > "> For #3, it seems simplest to have a config that requires clients
to
> > > > > support
> > > > > > a given message format version (or higher). For #2, it seems
like
> > > you'd
> > > > > > want clients to advertise their versions. That would be useful
for
> > > > > multiple
> > > > > > reasons."
> > > > > This kip offers the ability to block clients based on the protocol
they
> > > > > support. This should be independent of the message format upgrade.
Not
> > > all
> > > > > of the features or bugs are dependent on a message format and having
a
> > > > > message format dependency to block clients means we have to upgrade
to
> > > > > message.format and we cannot just say we've 1.1 brokers with 0.8.2
> > > message
> > > > > format and now we want to block all 0.8.x clients.
> > > > >
> > > > > min.api.version helps at the cluster level to say that all users
> > > required
> > > > > to upgrade clients to the at minimum need to speak the min.api.version
> > > and
> > > > > not tie to message.format because not all cases one wants to upgrade
> > > the
> > > > > message format and block the old clients.
> > > > >
> > > > >
> > > > > To Gwen's point, I think we should also return in the error message
> > > that
> > > > > the broker only supports min.api.version and above. So that users
can
> > > see a
> > > > > clear message and upgrade to a newer version.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > >
> > > > > On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote:
> > > > > > Hi Ying,
> > > > > >
> > > > > > The actual reasons are important so that people can evaluate
the KIP
> > > (and
> > > > > > vote). :) Thanks for providing a few more:
> > > > > >
> > > > > > (1) force users to check pointing in Kafka instead of zookeeper
> > > > > > (2) forbid an old go (sarama) client library which is known
to have
> > > some
> > > > > > serious bugs
> > > > > > (3) force kafka 1.x clients with the ability to roll back if
there's
> > > an
> > > > > > issue (unlike a message format upgrade)
> > > > > >
> > > > > > Relying on min.version seems like a pretty clunky way to achieve
the
> > > > > above
> > > > > > list. The challenge is that it's pretty difficult to do it in
a way
> > > that
> > > > > > works for clients across languages. They each add support for
new
> > > > > protocol
> > > > > > versions independently (it could even happen in a bug fix release).
> > > So,
> > > > > if
> > > > > > you tried to block Sarama in #2, you may block Java clients
too.
> > > > > >
> > > > > > For #3, it seems simplest to have a config that requires clients
to
> > > > > support
> > > > > > a given message format version (or higher). For #2, it seems
like
> > > you'd
> > > > > > want clients to advertise their versions. That would be useful
for
> > > > > multiple
> > > > > > reasons.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yingz@uber.com.invalid>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Ismael,
> > > > > > >
> > > > > > > Those are just examples. I think the administrators should
be able
> > > to
> > > > > block
> > > > > > > certain client libraries for whatever reason. Some other
possible
> > > > > reasons
> > > > > > > include, force users to check pointing in Kafka instead
of
> > > zookeeper,
> > > > > > > forbid an old go (sarama) client library which is known
to have
> > > some
> > > > > > > serious bugs.
> > > > > > >
> > > > > > > message.downconversion.enable does not solve our problems.
We are
> > > now
> > > > > > > planning to upgrade to message format V3, and force users
to
> > > upgrade to
> > > > > > > Kafka 1.x clients. With the proposed min.api.version setting,
in
> > > case
> > > > > of
> > > > > > > there is anything wrong, we can roll back the setting.
If we
> > > upgrade
> > > > > the
> > > > > > > file format, there is no way to rollback (Kafka doesn't
support
> > > > > downgrading
> > > > > > > message format).
> > > > > > >
> > > > > > > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <ismael@juma.me.uk>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Ying,
> > > > > > > >
> > > > > > > > It looks to me that all the examples given in the
KIP can be
> > > handled
> > > > > with
> > > > > > > > the existing "message.downconversion.enable" config
and by
> > > > > configuring
> > > > > > > the
> > > > > > > > message format to be the latest:
> > > > > > > >
> > > > > > > > 1. Kafka 8 / 9 / 10 consumer hangs when the message
contains
> > > message
> > > > > > > header
> > > > > > > > > ( KAFKA-6739 - Down-conversion fails for records
with headers
> > > > > > > RESOLVED  )
> > > > > > > > > 2. LZ4 is not correctly handled in Kafka 8 and
Kafka 9 (
> > > > > KAFKA-3160 -
> > > > > > > > > Kafka LZ4 framing code miscalculates header checksum
RESOLVED
> > > )
> > > > > > > > > 3. Performance penalty of converting message
format from V3 to
> > > V1
> > > > > or V2
> > > > > > > > > for the old consumers (KIP-31 - Move to relative
offsets in
> > > > > compressed
> > > > > > > > > message sets)
> > > > > > > >
> > > > > > > >
> > > > > > > > Am I missing something? Are there other examples that
are not
> > > > > related to
> > > > > > > > message conversion?
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng
> > > <yingz@uber.com.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi here,
> > > > > > > > >
> > > > > > > > > Please vote for
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> > > > > > > > >
> > > > > > > > > Thank you!
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
View raw message