activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Murphy" <bmurphy1...@gmail.com>
Subject Re: ActiveMQ + NMS = Stalled Message Queues
Date Fri, 29 Aug 2008 17:08:41 GMT
Ok, I'll give that a shot.  I just set my prefetch size to 1 in production.
 I'm going to monitor that for a bit and see what happens, but it already
stopped once.  I had not restarted the ActiveMQ process, so I restarted
everything.
Also, I'm planning to downgrade to the 4.x builds of ActiveMQ to see if that
makes a difference.

Bryan

On Fri, Aug 29, 2008 at 11:47 AM, Vadim Chekan <kot.begemot@gmail.com>wrote:

> Waiting in Reader from binary stream: that's expected. Another thread
> in dispatcher is normal too.
> I think you need to turn on debugging on the client side. See my
> article on how to do it:
>
> http://vchekan.blogspot.com/2008/08/tracing-in-nms-it-easy-to-turn-protocol.html
>
> Vadim.
>
>
> On Fri, Aug 29, 2008 at 8:01 AM, Bryan Murphy <bmurphy1976@gmail.com>
> wrote:
> > More information...
> > There were ~ 4,000 messages still pending in the queue.  I took a closer
> > look at the two active consumers under jconsole, the consumer that was
> > running had approximately 800 messages in it's
> > MessageCountAwaitingAcknowledge, the consumer that was idle had 0.
> >
> > I put a few new messages into the message queue, and this caused the idle
> > process to unblock and start processing messages.  Whatever this did, it
> > unstuck the idle process and I now have significantly less than 4,000
> > messages and it's continuing to tick downwards.
> >
> > It's pretty clear that ActiveMQ decided to stop sending messages to my
> > consumer.
> >
> > Bryan
> >
> > On Fri, Aug 29, 2008 at 9:40 AM, Bryan Murphy <bmurphy1976@gmail.com>
> wrote:
> >
> >> I changed my consumers to use client-acknowledge and immediately
> >> acknowledge the message upon receipt.  My handler does the following:
> >>
> >> void Handle(IMessage message)
> >> {
> >>   log message
> >>   acknowledge message
> >>   Thread.Sleep(random amount of time);
> >> }
> >>
> >> I bound the handler to consumer.Listen.  I started two processes last
> >> night, one that sleeps randomly from 0 to 1,000ms, and another that
> sleeps
> >> randomly from 0 to 30,000ms.
> >>
> >> This morning, when I got in, the process that sleeps for up to 1000ms is
> >> idle, and the process that sleeps for up to 30,000ms is still handling
> >> messages.
> >>
> >> I attached the debugger to the idle process and it has  three active
> >> threads.  One is in my code where I sit in a while(true) {
> Thread.Sleep(); }
> >> loop to prevent the application from exiting, and the other two are
> ActiveMQ
> >> threads.
> >>
> >> Thead appears to be blocked trying to read binary data from the tcp
> stream
> >> :
> >> * OpenWireBinaryReader.cs line 132
> >> * OpenWireFormat.cs line 165
> >> * TcpTransport.cs line 266
> >>
> >> Thread two is blocked waiting for an AutoResetEvent:
> >> * DispatchingThread.cs line 125
> >>
> >> I opened up jconsole and took a look at the ActiveMQ process but I'm no
> >> expert with that and couldn't find anything obviously wrong.  I'm at a
> loss
> >> what to do next.
> >>
> >> Thanks,
> >> Bryan
> >>
> >> On Thu, Aug 28, 2008 at 9:01 PM, Bryan Murphy <bmurphy1976@gmail.com
> >wrote:
> >>
> >>> I read more about calling acknowledge, and as far as I can tell,
> because I
> >>> was using transactional acknowledgement as well as session commit and
> >>> rollback, the call to acknowledge would have been a no-op.
> >>> Now, some of my transactions are in fact very long transactions.  We
> have
> >>> two sets of services, one that sits in our data center near our
> database,
> >>> and another one that sits out on Amazon's EC2 cloud processing audio
> and
> >>> video files and syncing them up to S3.  The service by our database
> handles
> >>> messages very quickly and never takes longer than a second (in theory),
> >>> however, it often sends out new messages as part of the transaction
> (which
> >>> I'll get to below).
> >>>
> >>> The services out on EC2 can take minutes to execute and hold the
> >>> transaction open the whole time.  It may very well be the case that
> ActiveMQ
> >>> thinks my consumers are slow and is throttling them.
> >>>
> >>> The database handlers fail more often than the EC2 handlers.  I found
> this
> >>> odd at first, until I added some performance metrics to our tracing.
>  It
> >>> turns out (on my local machine at least) that establishing a connection
> can
> >>> take upwards of 4.5 seconds.  We establish a new connection and a new
> >>> session every single time we send a message out.  Most of the handlers
> on
> >>> our database side send out at least one message, so the transactions
> are
> >>> held open for at least that long.
> >>>
> >>> So, it's obvious to me that there's no connection pooling going on and
> >>> reading the FAQ confirms this.  Assuming there was connection pooling
> was
> >>> clearly a bad assumption on my part.  We don't use spring internally,
> so we
> >>> have no pre-built components available to handle connection pooling.
>  For
> >>> now, I'm going to deal with it by simply moving away from transactional
> >>> message processing, and immediately acknowledging every message when it
> is
> >>> received.  I'll deal with the consequences of failing message handlers
> >>> later.
> >>>
> >>> I was also consuming messages in a very different fashion last time I
> used
> >>> a non-transactional method, so perhaps this change in combination with
> that
> >>> previous change will get my system into working order.  Once again,
> I'll
> >>> post back here with the results of these changes.
> >>>
> >>> If you notice any flaw in my reasoning, please do tell me. :)
> >>>
> >>> Thanks,
> >>> Bryan
> >>>
> >>>
> >
>
>
>
> --
> From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT
> is explicitly specified
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message