camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Marcus (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CAMEL-8221) No backpressure in aws-sqs Consumer
Date Tue, 17 Feb 2015 23:51:12 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322314#comment-14322314
] 

Steven Marcus edited comment on CAMEL-8221 at 2/17/15 11:50 PM:
----------------------------------------------------------------

I think blocking exchanges is Camel's implementation of back-pressure for polling consumers.

Have you tried using a seda component to block additional polls until in-flight exchanges
complete?
I only have experience doing this with the swf component:

from("aws-swf://activity").
to("seda:seda? blockWhenFull=true&size=2&concurrentConsumers=2&waitForTaskToComplete=Always&timeout=0")

SQS is a ScheduledBatchPollingConsumer so you will need to set maxMessagesPerPoll.
A value of 1 is probably right if you have "slow" consumers.

I don't know if there are any other implications of ScheduledBatchPollingConsumer vs. a simple
PollingConsumer so what I say here may not be effective with SQS. ymmv... GL!





was (Author: srnm):
I think blocking exchanges is Camel's implementation of back-pressure for polling consumers.

Have you tried using a seda component to block additional polls until in-flight exchanges
complete?
I only have experience doing this with the swf component:

from("aws-swf://activity").
to("seda:seda?concurrentConsumers=2&waitForTaskToComplete=Always&timeout=0")

SQS is a ScheduledBatchPollingConsumer so you will need to set maxMessagesPerPoll.
A value of 1 is probably right if you have "slow" consumers.

I don't know if there are any other implications of ScheduledBatchPollingConsumer vs. a simple
PollingConsumer so what I say here may not be effective with SQS. ymmv... GL!




> No backpressure in aws-sqs Consumer
> -----------------------------------
>
>                 Key: CAMEL-8221
>                 URL: https://issues.apache.org/jira/browse/CAMEL-8221
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-aws
>    Affects Versions: 2.14.1
>            Reporter: Dan Brown
>             Fix For: Future
>
>
> We're using a camel Consumer with an aws-sqs endpoint and running into issues with large
queues and slow consumers. The consumer jvm downloads messages very fast from sqs, onto the
jvm heap, even though the camel receive method processes each message very slowly (and Ack's
when done, with autoAck = false). The result of this is that the consumer jvm continually
fills its heap with an unbounded queue of incoming messages and eventually throws OOME.
> To avoid this failure, we're looking for a way to enable backpressure when using aws-sqs—e.g.
an on-heap bounded queue between the component fetching from sqs and the user-defined actor
processing the messages—but I don't see anything relevant in the config:
> - http://camel.apache.org/aws-sqs.html
> Looking at the code, I see that SqsConsumer subtypes ScheduledBatchPollingConsumer, which
subtypes ScheduledPollConsumer. To get backpressure, should it use something like EventDrivenPollingConsumer
somewhere, which uses a blocking queue to avoid the heap blowup above?
> - https://github.com/apache/camel/blob/camel-2.14.1/components/camel-aws/src/main/java/org/apache/camel/component/aws/sqs/SqsConsumer.java#L52-L57
> - https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledBatchPollingConsumer.java#L31-L34
> - https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledPollConsumer.java#L40-L43
> - https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/EventDrivenPollingConsumer.java#L35-L42



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message