activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vadim Chekan" <kot.bege...@gmail.com>
Subject Re: ActiveMQ + NMS = Stalled Message Queues
Date Fri, 29 Aug 2008 16:47:46 GMT
Waiting in Reader from binary stream: that's expected. Another thread
in dispatcher is normal too.
I think you need to turn on debugging on the client side. See my
article on how to do it:
http://vchekan.blogspot.com/2008/08/tracing-in-nms-it-easy-to-turn-protocol.html

Vadim.


On Fri, Aug 29, 2008 at 8:01 AM, Bryan Murphy <bmurphy1976@gmail.com> wrote:
> More information...
> There were ~ 4,000 messages still pending in the queue.  I took a closer
> look at the two active consumers under jconsole, the consumer that was
> running had approximately 800 messages in it's
> MessageCountAwaitingAcknowledge, the consumer that was idle had 0.
>
> I put a few new messages into the message queue, and this caused the idle
> process to unblock and start processing messages.  Whatever this did, it
> unstuck the idle process and I now have significantly less than 4,000
> messages and it's continuing to tick downwards.
>
> It's pretty clear that ActiveMQ decided to stop sending messages to my
> consumer.
>
> Bryan
>
> On Fri, Aug 29, 2008 at 9:40 AM, Bryan Murphy <bmurphy1976@gmail.com> wrote:
>
>> I changed my consumers to use client-acknowledge and immediately
>> acknowledge the message upon receipt.  My handler does the following:
>>
>> void Handle(IMessage message)
>> {
>>   log message
>>   acknowledge message
>>   Thread.Sleep(random amount of time);
>> }
>>
>> I bound the handler to consumer.Listen.  I started two processes last
>> night, one that sleeps randomly from 0 to 1,000ms, and another that sleeps
>> randomly from 0 to 30,000ms.
>>
>> This morning, when I got in, the process that sleeps for up to 1000ms is
>> idle, and the process that sleeps for up to 30,000ms is still handling
>> messages.
>>
>> I attached the debugger to the idle process and it has  three active
>> threads.  One is in my code where I sit in a while(true) { Thread.Sleep(); }
>> loop to prevent the application from exiting, and the other two are ActiveMQ
>> threads.
>>
>> Thead appears to be blocked trying to read binary data from the tcp stream
>> :
>> * OpenWireBinaryReader.cs line 132
>> * OpenWireFormat.cs line 165
>> * TcpTransport.cs line 266
>>
>> Thread two is blocked waiting for an AutoResetEvent:
>> * DispatchingThread.cs line 125
>>
>> I opened up jconsole and took a look at the ActiveMQ process but I'm no
>> expert with that and couldn't find anything obviously wrong.  I'm at a loss
>> what to do next.
>>
>> Thanks,
>> Bryan
>>
>> On Thu, Aug 28, 2008 at 9:01 PM, Bryan Murphy <bmurphy1976@gmail.com>wrote:
>>
>>> I read more about calling acknowledge, and as far as I can tell, because I
>>> was using transactional acknowledgement as well as session commit and
>>> rollback, the call to acknowledge would have been a no-op.
>>> Now, some of my transactions are in fact very long transactions.  We have
>>> two sets of services, one that sits in our data center near our database,
>>> and another one that sits out on Amazon's EC2 cloud processing audio and
>>> video files and syncing them up to S3.  The service by our database handles
>>> messages very quickly and never takes longer than a second (in theory),
>>> however, it often sends out new messages as part of the transaction (which
>>> I'll get to below).
>>>
>>> The services out on EC2 can take minutes to execute and hold the
>>> transaction open the whole time.  It may very well be the case that ActiveMQ
>>> thinks my consumers are slow and is throttling them.
>>>
>>> The database handlers fail more often than the EC2 handlers.  I found this
>>> odd at first, until I added some performance metrics to our tracing.  It
>>> turns out (on my local machine at least) that establishing a connection can
>>> take upwards of 4.5 seconds.  We establish a new connection and a new
>>> session every single time we send a message out.  Most of the handlers on
>>> our database side send out at least one message, so the transactions are
>>> held open for at least that long.
>>>
>>> So, it's obvious to me that there's no connection pooling going on and
>>> reading the FAQ confirms this.  Assuming there was connection pooling was
>>> clearly a bad assumption on my part.  We don't use spring internally, so we
>>> have no pre-built components available to handle connection pooling.  For
>>> now, I'm going to deal with it by simply moving away from transactional
>>> message processing, and immediately acknowledging every message when it is
>>> received.  I'll deal with the consequences of failing message handlers
>>> later.
>>>
>>> I was also consuming messages in a very different fashion last time I used
>>> a non-transactional method, so perhaps this change in combination with that
>>> previous change will get my system into working order.  Once again, I'll
>>> post back here with the results of these changes.
>>>
>>> If you notice any flaw in my reasoning, please do tell me. :)
>>>
>>> Thanks,
>>> Bryan
>>>
>>>
>



-- 
>From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT
is explicitly specified

Mime
View raw message