camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toli Kuznets <tkuzn...@marinsoftware.com>
Subject Re: ActiveMQ, Camel CBR routing issue
Date Wed, 01 Jan 2014 20:37:59 GMT
Claus,

Prefetch limit may be the culprit here. We have ours set to 1 since we
have fairly long-running jobs (sometimes hours), and our volumes are
fairly low (we have bursts of 100s or 1000s every hour, and then
lulls)

What would you recommend? Raising our prefetchLimit? Seems like that
goes against the advice of setting prefetchLimit=1 for long-running
low-volume jobs, as noted in the docs (ie "If you have very few
messages and each message takes a very long time to process you might
want to set the prefetch value to 1 so that a consumer is given one
message at a time")

thanks

On Wed, Jan 1, 2014 at 2:23 AM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
> Hi
>
> When its something with < 1000 then it sounds like a prefetch limit
> that affects this.
>
> See more on the AMQ web  site about that such as
> http://activemq.apache.org/what-is-the-prefetch-limit-for.html
>
> On Tue, Dec 31, 2013 at 10:52 PM, Toli Kuznets
> <tkuznets@marinsoftware.com> wrote:
>> Have a basic question about setting up AMQ and Camel context-based routing.
>> We added a Camel's content based routing for ActiveMQ message systems,
>> but the system stops throttling when the pending messages bellow a
>> specific threshold ( approx ~600)
>>
>> the CBR details
>>
>>             from("activemq:queue:Queue_In?concurrentConsumers=50&exchangePattern=InOut")
>>               .id("UniqueRouteId")
>>               .to("activemq:queue:Queue_Out?replyTo=Ack_Queue&concurrentConsumers=50&replyToType=Shared&requestTimeout=400000");
>>
>>
>> Another service is exposed to change the in-flight message count,
>> which will stop and remove the route using the unique id and re-create
>> the route with new value, which is working fine.
>>
>> When we send ~10K messages, all messages are routed properly ( only
>> the specified ( in concurrentConsumers) message will be delivered to
>> out queue at any given time), but when the pending message is below (
>> appox ~600), it will send only one message at a time even though the
>> concurrent message count is greater than one.  We tried to reduce the
>> memory limit (memoryUsage) in AMQ, but still if the pending message
>> goes below some threshold ( it reduced to ~50), it starts sending only
>> one message at time.
>>
>> We also tried to turn off the producer control flow setting in
>> ActiveMQ (<policyEntry queue=">" producerFlowControl="false"
>> memoryLimit="1mb">), but still it stops throttling if the pending
>> message is bellow same threshold
>>
>> We tried to add routing policy with higher max in-flight message count
>> ( e.g:- 50000) and resume percentage of max value ( e.g:- 50), it
>> seems to work fine for some period of time, but again it starts
>> sending one message at a time later.
>>
>> We are unable to find the relationship with ActiveMQ memory
>> parameters, producer flow controller and max in-flight message count
>> settings in Camel.
>>
>> Has anybody seen anything similar? I wouldn't be surprised if we are
>> doing something stupid, but we haven't been able to find documentation
>> on the relationship of all of these tunable parameters.
>>
>> It would be really great if you provide some ideas or documentation
>> for these settings.
>>
>> thanks (and happy new year!)
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io

Mime
View raw message