flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hilmi Yildirim (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2624) RabbitMQ source / sink should participate in checkpointing
Date Mon, 07 Sep 2015 15:04:45 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733814#comment-14733814
] 

Hilmi Yildirim commented on FLINK-2624:
---------------------------------------

Unfortunately, there is another problem. RabbitMQ delivers messages to the consumers and deletes
messages if they are acknowledged. RabbitMQ does deliver a message to only one consumer and
waits for the acknowledgement until the connection is lost, e.g. the consumer crashes. In
this case RabbitMQ sends the messages which are not acknowledged to other consumsers. For
example, consumer c received messages m1,m2 and m3. If a consumer has acknowledged only m1
without m2 and m3 and he crashes, then m2 and m3 will be delivered to other conumsers.

This means in our case: If a worker node crashes during the snapshotting without acknowledging
all processed messages, then the state of this worker node is useless. Furthermore, the messages
the consumer has acknowledged before he crashed, are deleted.

> RabbitMQ source / sink should participate in checkpointing
> ----------------------------------------------------------
>
>                 Key: FLINK-2624
>                 URL: https://issues.apache.org/jira/browse/FLINK-2624
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>    Affects Versions: 0.10
>            Reporter: Stephan Ewen
>            Assignee: Hilmi Yildirim
>
> The RabbitMQ connector does not offer any fault tolerance guarantees right now, because
it does not participate in the checkpointing.
> We should integrate it in a similar was as the {{FlinkKafkaConsumer}} is integrated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message