flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-10195) RabbitMQ Source With Checkpointing Doesn't Backpressure Correctly
Date Thu, 23 Aug 2018 17:46:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590591#comment-16590591
] 

Stephan Ewen commented on FLINK-10195:
--------------------------------------

Is that something that can be configured on RabbitMQ, for example a capacity bound on the
queue in the QueuingConsumer?

> RabbitMQ Source With Checkpointing Doesn't Backpressure Correctly
> -----------------------------------------------------------------
>
>                 Key: FLINK-10195
>                 URL: https://issues.apache.org/jira/browse/FLINK-10195
>             Project: Flink
>          Issue Type: Bug
>          Components: RabbitMQ Connector
>    Affects Versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0
>            Reporter: Luka Jurukovski
>            Priority: Major
>
> The connection between the RabbitMQ server and the client does not appropriately back
pressure when auto acking is disabled. This becomes very problematic when a downstream process
throttles the data processing to slower then RabbitMQ sends the data to the client.
> The difference in records ends up being stored in the flink's heap space, which grows
indefinitely (or technically to "Integer Max" Deliveries). Looking at RabbitMQ's metrics the
number of unacked messages looks like steadily rising saw tooth shape.
> Upon further invesitgation it looks like this is due to how the QueueingConsumer works,
messages are added to the BlockingQueue faster then they are being removed and processed,
resulting in the previously described behavior.
> This may be intended behavior, however this isn't explicitly obvious in the documentation
or any of the examples I have seen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message