Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
In-Reply-To: <1257d882-4ec0-4dc5-7ee6-7767da090e86@dueck.org>
References: <CAHwHRrWaAZFv83bH8t0_mneLoPZrbXNRo3sAM0H48ePE3-o5VA@mail.gmail.com>
 <CAAOfhrDd67R_357jzQs5p4ocBEQtayF40n_WrQwnLoLpZJ0pDw@mail.gmail.com>
 <CAOeJiJhvHKgP76YZrgv29WNeBBbumCW93agm8sM8HzGD5mTqpg@mail.gmail.com>
 <CANTWn9Nu5i9qNYCt+mCzDLJvb5=FH2tT-XRkqCwycjV0mzQL6w@mail.gmail.com>
 <CAOeJiJikE59rVsRYP58zbFpKCPEnyb-=S1Um_cCfSfSS1vvf2w@mail.gmail.com>
 <CACBUTrcAEmT51uZ_MwTNZiLkk6CiCRL6st-PqhNEHUM=v8c6Sw@mail.gmail.com>
 <CAAOfhrAWfHJxr9r+fM=YCT0MiP0KO8C9mKQTSpT8bausHKzFQw@mail.gmail.com>
 <CACBUTrdp04Cc64rBZqoVdoUyoVyJdqTtk4A4MJ6wjLqbuju4CA@mail.gmail.com>
 <CAFc58G88hpERh2Nf2J50OoFjCYXx2Mk0fOZS0BO5XQPqi8AYNg@mail.gmail.com>
 <CANTWn9Mijp2DGGG4vQFSx-1zDn8M3PBJd_nXAEC5k2xKTBj6Sw@mail.gmail.com>
 <CAFc58G-nhfHkn4BALwY2790NYpnT5i3aNdztisjag0ZXyMwWAg@mail.gmail.com>
 <CACBUTrfGf02aug_juwkGN-OKv3sLmQhUc18L5u+ndsu96b6vKg@mail.gmail.com>
 <CACBUTreNA7jn6mQW+yQ+31X+7Hsm_v3s_fV6pCVOHVS7OcLFuQ@mail.gmail.com>
 <CACBUTrcyTb6p2o1Fgb0hdQyifOxY_P8_UgiFwXNYNrC=dm6wDA@mail.gmail.com> <1257d882-4ec0-4dc5-7ee6-7767da090e86@dueck.org>
From: Apurva Mehta <apurva@confluent.io>
Date: Mon, 30 Jan 2017 11:39:44 -0800
Message-ID: <CACBUTrdRaWxNk8DydAzcWBgJaCxdJcEDeAxtjB0yVvu1OgfYgQ@mail.gmail.com>
Subject: Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging
To: dev@kafka.apache.org
Content-Type: multipart/alternative; boundary=001a1147c5ae3d6b60054754fa52
archived-at: Mon, 30 Jan 2017 19:39:52 -0000

--001a1147c5ae3d6b60054754fa52
Content-Type: text/plain; charset=UTF-8

>
> Eugen, moving your email to the main thread so that it doesn't get split.
>>
>> The `transaction.app.id` is a prerequisite for using transactional APIs.
>> And only messages wrapped inside transactions will enjoy idempotent
>> guarantees across sessions, and that too only when they employ a
>> consume-process-produce pattern.
>>
>
> Say I have a producer, producing messages into a topic and I only want to
> guarantee the producer cannot insert duplicates. In other words, there's no
> downstream consumer/processor to be worried about - which, when considering
> the correctness of the data only, is all I need for idempotent producers,
> as every message has a unique id (offset), so downstream processes can take
> care of exactly once processing by any number of means. (If you need
> transactional all-or-none behavior, which KIP-98 also addresses, that's of
> course a more complex story)
>
> I was under the impression that KIP-98 would fulfill above requirement,
> i.e. the prevention of duplicate inserts of the same message into a topic
> per producer, without using transactions, and guaranteed across tcp
> connections to handle producer/broker crashes and network problems.


The KIP-98 idempotent producer solution only protects against duplicates in
the stream when there are broker failures and network problems. For
instance, if a producer writes a message, and the leader commits and
replicates the message but dies before the acknowledgement is sent to the
client. Today, the client will resend the message which will be accepted by
the new leader, hence causing duplicates. Also, the offsets of the
duplicate messages in this case will be unique, so they can't be de-duped
downstream based on the offset.

If the client application itself dies, it needs to know which messages were
previously sent so that it doesn't resend them when it comes back online.
The proposed solution to this situation is to use transactional APIs and
the consume-process-produce pattern. If you do so, partially processed
previous inputs will be discarded, and processing will resume from the last
committed state.


>
> In other words, producers where the `transaction.app.id` is specified will
>> not enjoy idempotence across sessions unless their messages are
>> transactional. ie. that the sends  are wrapped between `beginTransaction`,
>> `sendOffsets`, and `commitTransaction`.
>>
>
> From the KIP-98 wiki and the design document, I understand that AppIDs,
> PIDs, and sequence numbers are enforced regardless of their being wrapped
> in a transaction or not. Is that not so?
>
>
The PID and sequence numbers are totally transparent to applications. If
you enable idempotent production, these will be created and managed by
Kafka.

AppIds only need to be specified if you use the four new transactional
APIs. This is enforced at runtime.

--001a1147c5ae3d6b60054754fa52--