kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Randall Hauch (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (KAFKA-6079) Idempotent production for source connectors
Date Mon, 29 Jan 2018 18:49:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Randall Hauch resolved KAFKA-6079.
    Resolution: Won't Fix

Marking this as WONTFIX since it is *already possible since AK 0.11* to configure the Connect
worker producers [to use idempotent delivery|https://kafka.apache.org/documentation/#upgrade_11_exactly_once_semantics]:

Idempotent delivery ensures that messages are delivered exactly once to a particular topic
partition during the lifetime of a single producer.

That eliminates duplicate events due to retries, but a failure of a Connector worker might
still mean records are written to Kafka but the worker fails before it can commit the latest
offsets; when it restarts, it begins from the last committed offsets and this may certainly
result in duplicate messages that were written before the failure.

See KAFKA-6080 for changing Connect to support exactly-once semantics for source connectors.

> Idempotent production for source connectors
> -------------------------------------------
>                 Key: KAFKA-6079
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6079
>             Project: Kafka
>          Issue Type: New Feature
>          Components: KafkaConnect
>            Reporter: Antony Stubbs
>            Priority: Major
> Idempotent production for source connection to reduce duplicates at least from retires.

This message was sent by Atlassian JIRA

View raw message