camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Brown <jdanbr...@gmail.com>
Subject Backpressure with aws-sqs Consumer?
Date Tue, 06 Jan 2015 00:18:19 GMT
Hello,

We're using a camel Consumer with an aws-sqs endpoint and running into
issues with large queues and slow consumers. The consumer jvm appears to
download messages very fast from sqs, onto the jvm heap, even though the
camel receive method processes each message very slowly (and Ack's when
done, with autoAck = false). The result of this is that the consumer jvm
continually fills its heap with an unbounded queue of incoming messages and
eventually throws OOME.

To avoid this failure, is there a way to enable backpressure when using
aws-sqs?—e.g. an on-heap bounded queue between the component fetching from
sqs and the user-defined actor processing the messages. I didn't see
anything relevant in the config:
- http://camel.apache.org/aws-sqs.html

And I didn't see any relevant jira issues:
-
https://issues.apache.org/jira/browse/CAMEL-5113?jql=project%20%3D%20CAMEL%20AND%20text%20~%20sqs

Looking at the code, I see that SqsConsumer subtypes
ScheduledBatchPollingConsumer, which subtypes ScheduledPollConsumer. To get
backpressure, should it use something like EventDrivenPollingConsumer
somewhere, which uses a blocking queue to avoid the heap blowup above?
-
https://github.com/apache/camel/blob/camel-2.14.1/components/camel-aws/src/main/java/org/apache/camel/component/aws/sqs/SqsConsumer.java#L52-L57
-
https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledBatchPollingConsumer.java#L31-L34
-
https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledPollConsumer.java#L40-L43
-
https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/EventDrivenPollingConsumer.java#L35-L42

Thanks,
Dan



--
View this message in context: http://camel.465427.n5.nabble.com/Backpressure-with-aws-sqs-Consumer-tp5761348.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Mime
View raw message