activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Murphy" <bmurphy1...@gmail.com>
Subject Re: ActiveMQ + NMS = Stalled Message Queues
Date Fri, 29 Aug 2008 14:40:59 GMT
I changed my consumers to use client-acknowledge and immediately acknowledge
the message upon receipt.  My handler does the following:

void Handle(IMessage message)
{
  log message
  acknowledge message
  Thread.Sleep(random amount of time);
}

I bound the handler to consumer.Listen.  I started two processes last night,
one that sleeps randomly from 0 to 1,000ms, and another that sleeps randomly
from 0 to 30,000ms.

This morning, when I got in, the process that sleeps for up to 1000ms is
idle, and the process that sleeps for up to 30,000ms is still handling
messages.

I attached the debugger to the idle process and it has  three active
threads.  One is in my code where I sit in a while(true) { Thread.Sleep(); }
loop to prevent the application from exiting, and the other two are ActiveMQ
threads.

Thead appears to be blocked trying to read binary data from the tcp stream :

* OpenWireBinaryReader.cs line 132
* OpenWireFormat.cs line 165
* TcpTransport.cs line 266

Thread two is blocked waiting for an AutoResetEvent:
* DispatchingThread.cs line 125

I opened up jconsole and took a look at the ActiveMQ process but I'm no
expert with that and couldn't find anything obviously wrong.  I'm at a loss
what to do next.

Thanks,
Bryan

On Thu, Aug 28, 2008 at 9:01 PM, Bryan Murphy <bmurphy1976@gmail.com> wrote:

> I read more about calling acknowledge, and as far as I can tell, because I
> was using transactional acknowledgement as well as session commit and
> rollback, the call to acknowledge would have been a no-op.
> Now, some of my transactions are in fact very long transactions.  We have
> two sets of services, one that sits in our data center near our database,
> and another one that sits out on Amazon's EC2 cloud processing audio and
> video files and syncing them up to S3.  The service by our database handles
> messages very quickly and never takes longer than a second (in theory),
> however, it often sends out new messages as part of the transaction (which
> I'll get to below).
>
> The services out on EC2 can take minutes to execute and hold the
> transaction open the whole time.  It may very well be the case that ActiveMQ
> thinks my consumers are slow and is throttling them.
>
> The database handlers fail more often than the EC2 handlers.  I found this
> odd at first, until I added some performance metrics to our tracing.  It
> turns out (on my local machine at least) that establishing a connection can
> take upwards of 4.5 seconds.  We establish a new connection and a new
> session every single time we send a message out.  Most of the handlers on
> our database side send out at least one message, so the transactions are
> held open for at least that long.
>
> So, it's obvious to me that there's no connection pooling going on and
> reading the FAQ confirms this.  Assuming there was connection pooling was
> clearly a bad assumption on my part.  We don't use spring internally, so we
> have no pre-built components available to handle connection pooling.  For
> now, I'm going to deal with it by simply moving away from transactional
> message processing, and immediately acknowledging every message when it is
> received.  I'll deal with the consequences of failing message handlers
> later.
>
> I was also consuming messages in a very different fashion last time I used
> a non-transactional method, so perhaps this change in combination with that
> previous change will get my system into working order.  Once again, I'll
> post back here with the results of these changes.
>
> If you notice any flaw in my reasoning, please do tell me. :)
>
> Thanks,
> Bryan
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message