flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Piotr Nowojski (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-6988) Add Apache Kafka 0.11 connector
Date Thu, 22 Jun 2017 17:09:00 GMT
Piotr Nowojski created FLINK-6988:

             Summary: Add Apache Kafka 0.11 connector
                 Key: FLINK-6988
                 URL: https://issues.apache.org/jira/browse/FLINK-6988
             Project: Flink
          Issue Type: Improvement
          Components: Kafka Connector
    Affects Versions: 1.3.1
            Reporter: Piotr Nowojski
            Assignee: Piotr Nowojski

Kafka 0.11 (it will be released very soon) add supports for transactions. Thanks to that,
Flink might be able to implement Kafka sink supporting "exactly-once" semantic. API changes
and whole transactions support is described in [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging].

The goal is to mimic implementation of existing BucketingSink. New KafkaProducer011 would

* upon creation begin transaction, store transaction identifiers into the state and would
write all incoming data to output topic using that transaction
* on `snapshotState` call, it would flush the data and write in state information that current
transaction is pending to be committed
* on `notifyCheckpointComplete` we would commit this pending transaction
* in case of crash between `snapshotState` and `notifyCheckpointComplete` we either abort
this pending transaction (if not every participant successfully saved the snapshot) or restore
and commit it. 

This message was sent by Atlassian JIRA

View raw message