kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Klijs <gerard.kl...@dizzit.com>
Subject Re: Message sent ordering guarantees
Date Thu, 01 Sep 2016 09:02:08 GMT
For async you could set ack to -1, but it keep slow, because it has to wait
for the broker(s), to know it is received in order, before sending the next
one, you need this if order is very important. When sending async the order
gets changed in case the leader becomes temporarily unavailable and other
errors, because data is send correct to the new leader, before it fails on
the old leader, and is send to the new leader.
Depending on your use case you have different options to handle this, for
example with using timestamps in the messages, so you know which event
happened first, bust this wil not work when you use compaction.

On Thu, Sep 1, 2016 at 6:19 AM 郭旭 <guoxu1231@gmail.com> wrote:

> Hi Kafka Experts,
>
> (Sorry to send this question to DEV group, but it seems that I can not find
> related document in user manual.)
>
> For official document ,I can find message sent guarantee as below.
> For *sync producer*, I think it is true but sync sent are very slow.(about
> 408 message per second if ack = all, 1000 message per second if ack = 1).
>
> batch and async sent could satisfy our throughput requirement, but I'm not
> sure if message sent ordering are guaranteed in *async *style.
>
> For some critical application, for example( replicate mysql binlog to kafka
> distributed committed log), binlog ordering are important(partitioned by
> database/table/PK). throughput also important.
>
> If I use async producer, partition the binlog by table and send them in
> batch. Is it safe for binlog ordering for a single table?
>
> Will async producer guarantee the send ordering?
>
>
> Regards
> Shawn
>
> Guarantees <http://kafka.apache.org/documentation.html#intro_guarantees>At
> a high-level Kafka gives the following guarantees:
>
>    - Messages sent by a producer to a particular topic partition will be
>    appended in the order they are sent. That is, if a message M1 is sent by
>    the same producer as a message M2, and M1 is sent first, then M1 will
> have
>    a lower offset than M2 and appear earlier in the log.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message