kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apurva Mehta <apu...@confluent.io>
Subject Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting
Date Sat, 19 Aug 2017 00:18:14 GMT
Thanks Jason and Ismael.

The message format problem is an acute one: if we enable idempotence by
default, the UnsupportedVersionException when writing to topics with the
older message format would mean that our prescribed upgrade steps would not
work. I have detailed the problems and the solutions on this page (linked
to from the wiki):

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Dealing+with+older+message+formats+when+idempotence+is+enabled

It is worth discussing the solution to the problem proposed there. If it is
conceptually sound, it doesnt' seem too hard to implement.

As far as the other problem of the spurious OutOfOrderSequence problem, I
have documented a proposed solution here:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Solving+the+problem+of+spurious+OutOfOrderSequence+errors

This solution is a bit more involved in terms of effort.

I think we cannot make the idempotent producer the default unless we solve
the message format compatibility problem. I would also prefer to solve the
second problem before making idempotence the default.

I would be interested to hear everyone's thoughts on the two solutions
proposed above.

Thanks,
Apurva

On Fri, Aug 18, 2017 at 9:24 AM, Jason Gustafson <jason@confluent.io> wrote:

> >
> >  so this change will break client backward compatibility while connecting
> > to 0.10.X brokers.
> >  users need to change producer default settings while connecting older
> > brokers.
>
>
> At the moment, I think the answer is yes. The old broker will not support
> the InitProducerId request, so the producer will immediately fail. Similar
> to the handling of old message formats mentioned above, we probably need to
> change this so that we just revert to old producer semantics if the broker
> can't support idempotence.
>
> -Jason
>
>
> On Fri, Aug 18, 2017 at 8:48 AM, Manikumar <manikumar.reddy@gmail.com>
> wrote:
>
> > >
> > > 3. The message format requirement is a good point. This should be
> > mentioned
> > > in the compatibility section. Users who are still using the old message
> > > format will get an error after the upgrade, right?
> > >
> >
> >  so this change will break client backward compatibility while connecting
> > to 0.10.X brokers.
> >  users need to change producer default settings while connecting older
> > brokers.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message