activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: Slow consumers slow down fast consumer queue requests
Date Mon, 19 Oct 2009 07:00:20 GMT
temporary destinations add additional overhead because to be compliant  
with the JMS specification, a messaging system needs to validate that  
the temporary destination exists before sending every message. To  
avoid slowing delivery of messages to a crawl to a temporary queue,  
ActiveMQ sends advisory messages to every client on creation/deletion  
of a temporary destination. Hence if you are constantly creating/ 
destroying temporary destinations, you are creating a lot of  
additional chatter.

Not ideal - but at least this should explain what is happening. You  
can disable advisory messages - see http://activemq.apache.org/advisory-message.html 
.

On 19 Oct 2009, at 01:17, mquestioner wrote:

>
> Yeah... by reusing the same temporary queues, the problem goes away.  
> (We used
> 1 temp queue per request because the requests were originated at  
> different
> threads. We changed to use a fixed set of temp queues, listeners,  
> and local
> dispatching to work around.)
>
> But this seems just fix the symptoms. If supposedly, one has many  
> clients
> come and go and each of them would have to create at least some  
> queues; the
> system would still be slowed down. The bottomline is that creating  
> temp
> queues should not slow down others.
>
>
> aortiz wrote:
>>
>> You should not create the session and temporary queue for every  
>> request.
>> You should create them when you create the client [1] (or at least  
>> cache
>> them). These operations are expensive and require remote calls to the
>> broker.
>>
>> The slowdown of faster clients may be due to Producer Flow Control  
>> [2],
>> which is enabled by default.
>>
>> -Andres
>> [1]
>> http://activemq.apache.org/how-should-i-implement-request-response-with-jms.html
>> [2] http://activemq.apache.org/producer-flow-control.html
>>
>>
>> mquestioner wrote:
>>>
>>> Thanks. Will try.
>>>
>>> More detail about the client is that when we send a queue request,  
>>> we do
>>>
>>> - create a Session
>>> - create a TemporaryQueue
>>>
>>> I wonder if these two will be broadcasting something to the other  
>>> clients
>>> and cause the backlog. Shouldn't be, but...
>>>
>>>
>>>
>>> rajdavies wrote:
>>>>
>>>> Does this behaviour change if you use the latest snapshot ?
>>>>
>>>> thanks,
>>>>
>>>> Rob
>>>> On 20 Jul 2009, at 03:51, mquestioner wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> We have a case where a consumer connected through a slow  
>>>>> connection
>>>>> would
>>>>> slow down the fast consumer. I have already
>>>>> http://activemq.apache.org/slow-consumers.html and
>>>>> http://activemq.apache.org/slow-consumer-handling.html However,
>>>>> there are
>>>>> specific advises do not seem applicable to our scenario.
>>>>>
>>>>> Out set up looks like:
>>>>> The server
>>>>>  Constantly broadcasting non-persistent messages over a topic (a)
>>>>>  Listen on a number of persistent and non-persistent queues to
>>>>> process
>>>>> messages asynchronously (Listener) (b)
>>>>>    Reply to the temporary queue of the request (c)
>>>>>
>>>>> The client
>>>>>  Listen to the broadcast messages in background (d)
>>>>>  Make request to the queues (e)
>>>>>     Wait for reply (f)
>>>>>
>>>>> When the clients are on fast connection, everything is fine.
>>>>> However, once a
>>>>> slow connecting client joins, we observe:
>>>>>
>>>>> 1) (e) and (f) of the fast client become extremely sluggish  
>>>>> instantly.
>>>>>
>>>>> 2) Timestamp logging of (b) and (c) shows that the server still
>>>>> takes very
>>>>> little time to process each request, ruling out problems such as  
>>>>> DB
>>>>> bottleneck. On the other hand, the messaging waiting time (time at
>>>>> the entry
>>>>> of the listner - msg.getJMStimestamp()) increases by 50-100X
>>>>>
>>>>> 3) Either killing the slow connecting client or stop broadcasting
>>>>> message
>>>>> will have things return to normal
>>>>>
>>>>> The strange thing is that, according to
>>>>> http://activemq.apache.org/slow-consumers.html, ``A slow  
>>>>> consumer is
>>>>> not
>>>>> really an issue with queues.'' And that the problem occurs almost
>>>>> instantly
>>>>> after the slow connection is established, it seems too short for  
>>>>> any
>>>>> memory
>>>>> usage built-up; and indeed no such problem is observed. We also  
>>>>> try
>>>>> twisting
>>>>> ActiveMQ broker and consumer settings like prefetchsize=2000,
>>>>> dedicated task
>>>>> runner = true/false,  increase memoryLimit/memoryUsage,
>>>>> optimizedDispatch=true. None seems to solve the problem.
>>>>>
>>>>> What would be the causes and solutions for these scenario?
>>>>>
>>>>> (The ActiveMQ site's suggestions of discarding old messages would
>>>>> create
>>>>> stale data problems for our application.)
>>>>>
>>>>> Thanks a lot!
>>>>>
>>>>>
>>>>> -- 
>>>>> View this message in context:
>>>>> http://www.nabble.com/Slow-consumers-slow-down-fast-consumer-queue-requests-tp24563120p24563120.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
> -- 
> View this message in context: http://www.nabble.com/Slow-consumers-slow-down-fast-consumer-queue-requests-tp24563120p25951782.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/






Mime
View raw message