kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From apurvam <...@git.apache.org>
Subject [GitHub] kafka pull request #2735: KAFKA-4815 : Add idempotent producer semantics
Date Fri, 24 Mar 2017 21:26:47 GMT
GitHub user apurvam opened a pull request:


    KAFKA-4815 : Add idempotent producer semantics

    This is from the KIP-98 proposal. 
    The main points of discussion surround the correctness logic, particularly the Log class
where incoming entries are validated and duplicates are dropped, and also the producer error
handling to ensure that the semantics are sound from the users point of view.
    There is some subtlety in the idempotent producer semantics. This patch only guarantees
idempotent production upto the point where an error has to be returned to the user. Once we
hit a such a non-recoverable error, we can no longer guarantee message ordering nor idempotence
without additional logic at the application level.
    In particular, if an application wants guaranteed message order without duplicates, then
it needs to do the following on the error callback:
    # Close the producer so that no queued batches are sent. This is important for guaranteeing
    # Read the tail of the log to inspect the last message committed. This is important for
avoiding duplicates.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/confluentinc/kafka exactly-once-idempotent-producer

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2735
commit 83f5b2937d368efa768be6d7df1f57d99425a252
Author: fpj <fpj@apache.org>
Date:   2016-10-10T16:19:52Z

    Add broker producer id mapping support

commit 37eac83364a67eebf0e1d96552d9fe864baa3d76
Author: Guozhang Wang <wangguoz@gmail.com>
Date:   2017-02-14T19:12:58Z

    KEOS: idempotent producer pid generation (#126)
    Add the transaction coordinator for pid generation and management.

commit 6d918d0b44302aa81627f3f2748756356614f228
Author: hachikuji <jason@confluent.io>
Date:   2017-02-23T18:59:46Z

    Minor fixes for test failures and consistency (#133)

commit d1868602be8b3e81d98644d673a73b9f730ad586
Author: Jason Gustafson <jason@confluent.io>
Date:   2017-03-01T22:40:30Z

    Fix headers, new checkstyle rules, and some breakage in KafkaApis

commit f2c01a71508cd5c1eaf8c03a372f7dc308af3cf2
Author: hachikuji <jason@confluent.io>
Date:   2017-03-07T00:23:11Z

    Avoid removal of non-expired PIDs when log cleaning (#138)

commit 31ff08eaf704b9e4fe6ee492277d981b47e8016c
Author: Apurva Mehta <apurva.1618@gmail.com>
Date:   2017-03-09T01:50:01Z

    Client side implementation of the idempotent producer. (#129)

commit 439f284fd4efa56449d43f2fb5e14614689c7b80
Author: Apurva Mehta <apurva@confluent.io>
Date:   2017-03-09T21:23:55Z

    Fix build errors due to rebase

commit 0a81a40258da29ed62caf745e7c1895ca7eead4c
Author: hachikuji <jason@confluent.io>
Date:   2017-03-10T17:50:01Z

    PIDs should be expired according to the transactional id expiration setting (#139)

commit 2643e5b6bb6ad6b3d1cd6b15073a92396be5693b
Author: Apurva Mehta <apurva@confluent.io>
Date:   2017-03-15T18:45:07Z

    Fix build errors due to rebase

commit 848137a28b02cd085605e92726d0457b0153e1c4
Author: Apurva Mehta <apurva@confluent.io>
Date:   2017-03-16T18:43:29Z

    Remove depependence of MemoryRecordsBuilder on TransactionState

commit 15d23b65166cff6dbd7be0e925ff9ec811d7baac
Author: hachikuji <jason@confluent.io>
Date:   2017-03-23T22:30:39Z

    A few minor improvements (#150)

commit fafffa8f2445e2a559f24e5286c948c616287f9a
Author: Apurva Mehta <apurva.1618@gmail.com>
Date:   2017-03-24T20:02:48Z

    Reset transaction state on all irrecoverable exceptions (#153)

commit 637f864887e341c51f4fd7192016d2afc45bcf1e
Author: Apurva Mehta <apurva@confluent.io>
Date:   2017-03-24T21:08:29Z

    Fix build errors due to rebase


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message