flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hilmi Yildirim (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2624) RabbitMQ source / sink should participate in checkpointing
Date Mon, 07 Sep 2015 14:10:45 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733753#comment-14733753
] 

Hilmi Yildirim commented on FLINK-2624:
---------------------------------------

No, it is not possible to set read offsets. Unlike Kafka, RabbitMQ deletes messages  after
they are acklowledged und does not keep them in the queue. Therefore, the snapshotting has
to be done differently. My approach is to keep the message ids without acknowleding them.
If the snapshotting is performed, then the messages are acknowledged and will be deleted from
the queue.

The problem is that the worker node can crash during the acknowledge phase. Therefore, the
message ids have to be stored in an internal state before acknowledging them. If the acknowledgement
is finished, the ids will be removed from the internal state.

> RabbitMQ source / sink should participate in checkpointing
> ----------------------------------------------------------
>
>                 Key: FLINK-2624
>                 URL: https://issues.apache.org/jira/browse/FLINK-2624
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>    Affects Versions: 0.10
>            Reporter: Stephan Ewen
>            Assignee: Hilmi Yildirim
>
> The RabbitMQ connector does not offer any fault tolerance guarantees right now, because
it does not participate in the checkpointing.
> We should integrate it in a similar was as the {{FlinkKafkaConsumer}} is integrated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message