camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Newsham <jnews...@referentia.com>
Subject Re: jms concurrentConsumers is not competing consumers
Date Tue, 07 Feb 2012 00:22:21 GMT

The MEP is InOut ("request/reply").  There are two participants, a 
consumer and a producer.  We need concurrent handling on the consumer 
(receiver, request processor) side.  From my reading of the 
documentation, the "concurrentConsumers" option should provide 
concurrency for the consumer side, and the "threads()" dsl should 
provide concurrency on the producer side.

Although my use case is InOut (not InOnly), I agree with your statement 
"concurrentConsumers should do it".  It's mostly doing it, but there is 
a critical liveness issue as I have described.

Regarding the route config, see my other email from a few hours ago.  It 
has full Java code for running a server and client demo of the problem, 
including initialization of routes.

Thanks,
Jim


On 2/6/2012 1:58 PM, David Karlsen wrote:
> In the original mail you stated "option on our jms consumer endpoint
> so that we can handle numerous InOut requests" which I read as
> Producer with InOut MEP - and from the docs I read that should use the
> threads options.
> If you only have incoming (e.g. InOnly) then I read the docs as
> concurrentConsumers should do it.
>
> Maybe you could show the route config?
>
> 2012/2/6 Jim Newsham<jnewsham@referentia.com>:
>> Thanks David for the suggestion, but the documentation says that this option
>> is for concurrently handling replies.  I am looking to concurrently handle
>> requests.  Regardless of the documentation, I just gave this a try and it
>> does not fix the issue.
>>
>> Thanks again,
>> Jim
>>
>> On 2/6/2012 10:01 AM, David Karlsen wrote:
>>> Read the section on "Request-reply over JMS" - you're looking for the
>>> threads option.
>>>
>>> 2012/2/6 Jim Newsham<jnewsham@referentia.com>:
>>>> Hi Ngoc Hai,
>>>>
>>>> I'm not talking at all about the producer side where replies are
>>>> processed,
>>>> but the consumer side where requests are being received and processed.
>>>>   Incoming requests are not being dispatched to idle threads; instead,
>>>> they
>>>> are backing up behind a single (non-idle) thread.
>>>>
>>>> To me, this looks like a bug.  I am interested in hearing the opinions of
>>>> the community on this point, and I'm willing to file a JIRA issue.
>>>>
>>>> Thanks,
>>>> Jim
>>>>
>>>> On 2/3/2012 6:32 PM, Pham Ngoc Hai wrote:
>>>>> Hi Jim,
>>>>> Since it's InOut queue, the reply is processed by a single thread as
>>>>> stated in here: http://camel.apache.org/jms.html no matter how many
>>>>> concurrentConsumer you set.
>>>>> Regards,
>>>>> Ngoc Hai.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>>   From: Jim Newsham<jnewsham@referentia.com>
>>>>> To: users@camel.apache.org
>>>>> Sent: Saturday, February 4, 2012 10:55 AM
>>>>> Subject: jms concurrentConsumers is not competing consumers
>>>>>
>>>>>
>>>>> We are using Camel 2.9.0.  We are using the "concurrentConsumers=n"
>>>>> option
>>>>> on our jms consumer endpoint so that we can handle numerous InOut
>>>>> requests
>>>>> concurrently.  In Camel documentation, this option is described as
>>>>> supporting competing consumers, which I interpret to mean a pool of
>>>>> threads
>>>>> which are _competing_ to handle incoming requests.  I expect that
>>>>> whenever a
>>>>> request comes in, that request will be processed by some idle thread
>>>>> from
>>>>> the thread pool (if there are any).  However, this is not that behavior
>>>>> we
>>>>> are observing.  Instead, I observe some requests not being processed
>>>>> immediately (or for quite some time), despite having numerous idle
>>>>> consumer
>>>>> threads, and this is causing a severe liveness issue for our
>>>>> application.
>>>>>
>>>>> Let me describe a simple scenario.  Assume we have a single InOut jms
>>>>> queue with concurrentConsumers=3, and the following stream of messages
>>>>> (and
>>>>> required processing time):
>>>>>
>>>>> Message #0 - 1h
>>>>> Message #1 - 1s
>>>>> Message #2 - 1s
>>>>> Message #3 - 1s
>>>>> Message #4 - 1s
>>>>> Message #5 - 1s
>>>>> Message #6 - 1s
>>>>> Message #7 - 1s
>>>>> Message #8 - 1s
>>>>> Message #9 - 1s
>>>>> Message #10 - 1s
>>>>>
>>>>> Because Message #0 requires 1 hour to process, it will occupy one
>>>>> concurrent consumer thread for 1 hour.  Nevertheless, we expect the
>>>>> remaining 2 threads should be able to handle the remaining 9 messages
>>>>> very
>>>>> quickly.  Instead, messages #3, #6, #9 are not processed until Message
>>>>> #0
>>>>> completes processing -- an hour later.  Based on this observation, it
>>>>> appears that messages are dispatched to queues owned by individual
>>>>> threads,
>>>>> in round-robin fashion, instead of having threads compete to consume
>>>>> messages from a single queue.  When some messages can take a long time
>>>>> to
>>>>> process, it can mean that messages can queue up and go unprocessed for
a
>>>>> very long time, even when there are free threads.
>>>>>
>>>>> As you can see, this is not "competing consumers".  Is the above
>>>>> behavior
>>>>> considered a bug?  Are there any suggested workarounds?
>>>>>
>>>>> Thanks!
>>>>> Jim
>>>>
>>>
>
>


Mime
View raw message