camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pham Ngoc Hai <pngoc...@yahoo.com>
Subject Re: jms concurrentConsumers is not competing consumers
Date Wed, 22 Feb 2012 10:19:12 GMT
Hi Jim,
Beside Babak's reply, if you still want the competing behavior, you can try the asyncConsumer
option on the consumer, be warned about the message's order, I have not tested that myself
though.
Regards,
Ngoc Hai.



________________________________
 From: Jim Newsham <jnewsham@referentia.com>
To: users@camel.apache.org 
Sent: Wednesday, February 15, 2012 8:15 AM
Subject: Re: jms concurrentConsumers is not competing consumers
 

Well, I haven't gotten a satisfactory response to this inquiry.  Really looks like a bug
to me.  Any Camel contributors care to comment?

Thanks,
Jim

On 2/6/2012 2:22 PM, Jim Newsham wrote:
> 
> The MEP is InOut ("request/reply").  There are two participants, a consumer and a producer. 
We need concurrent handling on the consumer (receiver, request processor) side.  From my
reading of the documentation, the "concurrentConsumers" option should provide concurrency
for the consumer side, and the "threads()" dsl should provide concurrency on the producer
side.
> 
> Although my use case is InOut (not InOnly), I agree with your statement "concurrentConsumers
should do it".  It's mostly doing it, but there is a critical liveness issue as I have described.
> 
> Regarding the route config, see my other email from a few hours ago.  It has full Java
code for running a server and client demo of the problem, including initialization of routes.
> 
> Thanks,
> Jim
> 
> 
> On 2/6/2012 1:58 PM, David Karlsen wrote:
>> In the original mail you stated "option on our jms consumer endpoint
>> so that we can handle numerous InOut requests" which I read as
>> Producer with InOut MEP - and from the docs I read that should use the
>> threads options.
>> If you only have incoming (e.g. InOnly) then I read the docs as
>> concurrentConsumers should do it.
>> 
>> Maybe you could show the route config?
>> 
>> 2012/2/6 Jim Newsham<jnewsham@referentia.com>:
>>> Thanks David for the suggestion, but the documentation says that this option
>>> is for concurrently handling replies.  I am looking to concurrently handle
>>> requests.  Regardless of the documentation, I just gave this a try and it
>>> does not fix the issue.
>>> 
>>> Thanks again,
>>> Jim
>>> 
>>> On 2/6/2012 10:01 AM, David Karlsen wrote:
>>>> Read the section on "Request-reply over JMS" - you're looking for the
>>>> threads option.
>>>> 
>>>> 2012/2/6 Jim Newsham<jnewsham@referentia.com>:
>>>>> Hi Ngoc Hai,
>>>>> 
>>>>> I'm not talking at all about the producer side where replies are
>>>>> processed,
>>>>> but the consumer side where requests are being received and processed.
>>>>>   Incoming requests are not being dispatched to idle threads; instead,
>>>>> they
>>>>> are backing up behind a single (non-idle) thread.
>>>>> 
>>>>> To me, this looks like a bug.  I am interested in hearing the opinions
of
>>>>> the community on this point, and I'm willing to file a JIRA issue.
>>>>> 
>>>>> Thanks,
>>>>> Jim
>>>>> 
>>>>> On 2/3/2012 6:32 PM, Pham Ngoc Hai wrote:
>>>>>> Hi Jim,
>>>>>> Since it's InOut queue, the reply is processed by a single thread
as
>>>>>> stated in here: http://camel.apache.org/jms.html no matter how many
>>>>>> concurrentConsumer you set.
>>>>>> Regards,
>>>>>> Ngoc Hai.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ________________________________
>>>>>>   From: Jim Newsham<jnewsham@referentia.com>
>>>>>> To: users@camel.apache.org
>>>>>> Sent: Saturday, February 4, 2012 10:55 AM
>>>>>> Subject: jms concurrentConsumers is not competing consumers
>>>>>> 
>>>>>> 
>>>>>> We are using Camel 2.9.0.  We are using the "concurrentConsumers=n"
>>>>>> option
>>>>>> on our jms consumer endpoint so that we can handle numerous InOut
>>>>>> requests
>>>>>> concurrently.  In Camel documentation, this option is described
as
>>>>>> supporting competing consumers, which I interpret to mean a pool
of
>>>>>> threads
>>>>>> which are _competing_ to handle incoming requests.  I expect that
>>>>>> whenever a
>>>>>> request comes in, that request will be processed by some idle thread
>>>>>> from
>>>>>> the thread pool (if there are any).  However, this is not that behavior
>>>>>> we
>>>>>> are observing.  Instead, I observe some requests not being processed
>>>>>> immediately (or for quite some time), despite having numerous idle
>>>>>> consumer
>>>>>> threads, and this is causing a severe liveness issue for our
>>>>>> application.
>>>>>> 
>>>>>> Let me describe a simple scenario.  Assume we have a single InOut
jms
>>>>>> queue with concurrentConsumers=3, and the following stream of messages
>>>>>> (and
>>>>>> required processing time):
>>>>>> 
>>>>>> Message #0 - 1h
>>>>>> Message #1 - 1s
>>>>>> Message #2 - 1s
>>>>>> Message #3 - 1s
>>>>>> Message #4 - 1s
>>>>>> Message #5 - 1s
>>>>>> Message #6 - 1s
>>>>>> Message #7 - 1s
>>>>>> Message #8 - 1s
>>>>>> Message #9 - 1s
>>>>>> Message #10 - 1s
>>>>>> 
>>>>>> Because Message #0 requires 1 hour to process, it will occupy one
>>>>>> concurrent consumer thread for 1 hour.  Nevertheless, we expect
the
>>>>>> remaining 2 threads should be able to handle the remaining 9 messages
>>>>>> very
>>>>>> quickly.  Instead, messages #3, #6, #9 are not processed until Message
>>>>>> #0
>>>>>> completes processing -- an hour later.  Based on this observation,
it
>>>>>> appears that messages are dispatched to queues owned by individual
>>>>>> threads,
>>>>>> in round-robin fashion, instead of having threads compete to consume
>>>>>> messages from a single queue.  When some messages can take a long
time
>>>>>> to
>>>>>> process, it can mean that messages can queue up and go unprocessed
for a
>>>>>> very long time, even when there are free threads.
>>>>>> 
>>>>>> As you can see, this is not "competing consumers".  Is the above
>>>>>> behavior
>>>>>> considered a bug?  Are there any suggested workarounds?
>>>>>> 
>>>>>> Thanks!
>>>>>> Jim
>>>>> 
>>>> 
>> 
>> 
> 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message