flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elias Levy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-4027) FlinkKafkaProducer09 sink can lose messages
Date Mon, 06 Jun 2016 21:08:20 GMT
Elias Levy created FLINK-4027:

             Summary: FlinkKafkaProducer09 sink can lose messages
                 Key: FLINK-4027
                 URL: https://issues.apache.org/jira/browse/FLINK-4027
             Project: Flink
          Issue Type: Bug
          Components: Kafka Connector
    Affects Versions: 1.0.3
            Reporter: Elias Levy
            Priority: Critical

The FlinkKafkaProducer09 sink appears to not offer at-least-once guarantees.

The producer is publishing messages asynchronously.  A callback can record publishing errors,
which will be raised when detected.  But as far as I can tell, there is no barrier to wait
for async errors from the sink when checkpointing or to track the event time of acked messages
to inform the checkpointing process.

If a checkpoint occurs while there are pending publish requests, and the requests return a
failure after the checkpoint occurred, those message will be lost as the checkpoint will consider
them processed by the sink.

This message was sent by Atlassian JIRA

View raw message