kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting
Date Sat, 12 Aug 2017 09:32:40 GMT
Hi all,

I will send a more detail email later, some quick comments:

1. It's unlikely that defaults will suit everyone. I think the question is:
what is the most likely configuration for a typical Kafka user _today_?
Kafka's usage is growing well beyond its original use cases and correctness
is often more important than achieving the maximum possible performance.
Large scale users have the time, knowledge and resources to tweak settings
so that they can achieve the latter. 1.0.0 is a good time to be thinking
about this.

2. This KIP focuses on producer configs. A few people (in the discussion
thread and offline) have asked whether the default min.insync.replicas
should be changed as well. I think that's a fair question and we should
address it in the KIP.

3. It is true that the current configs are generally too complicated. The
notion of "profiles" has been raised previously where one could state their
intent and the configs would be set to match. Becket suggested a semantics
config (exactly_once, at_most_once, at_least_once), which is similar to an
existing Kafka Streams config. The user could also specify if they would
like to optimise for latency or throughput, for example (this has been
raised before). I think we should pursue these ideas in a separate KIP.

One comment inline.

On Sat, Aug 12, 2017 at 1:26 AM, Becket Qin <becket.qin@gmail.com> wrote:

> From users' perspective, when idempotence=true and
> max.in.flight.requests.per.connection > 0, ideally what acks=1 should
> really mean is that "as long as there is no hardware failure, my message is
> sent exactly once".


I don't understand this statement. Hardware failures are common and we
should be striving for defaults that work correctly under failure.

Ismael

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message