activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.
Date Wed, 11 Mar 2015 01:19:08 GMT
You should be able to confirm that the prefetch buffers are empty by
inspecting the JMX MBeans on the broker.  Look at the consumers for the
destination, and for each one look at its DispatchedQueueSize attribute.

Keep in mind that slow consumers are identified *ONLY* if you configure one
of the abort strategies.  If you didn't set that up, don't expect any slow
consumer identification log lines.  And if you did, I've never seen a
situation where a consumer went slow and a log line didn't happen (using
the SlowConsumerAbortStrategy; I haven't used SlowAckConsumerAbortStrategy
and can't vouch for it); we get those log lines pretty frequently.  So if
you're not seeing broker-side log lines about consumers being identified as
slow and then aborted, I'd bet it's simply not happening.

On Tue, Mar 10, 2015 at 7:10 PM, Kevin Burton <burton@spinn3r.com> wrote:

> The broker.  I’ll assume the prefetch brokers are empty. I’m looking into
> debugging that now but I don’t have tools to introspect.
>
> The broker has thousands of messages.
>
> I just confirmed that a restart DOES improve the situation.
>
> It’s possible that they’re being marked as slow consumers but not *logged*
> as such so I’m trying to use JMX to dump the sessions.
>
> On Tue, Mar 10, 2015 at 5:58 PM, Tim Bain <tbain@alumni.duke.edu> wrote:
>
> > Are the messages getting hung up in the broker or in the client?  (Do the
> > consumers have empty or full prefetch buffers?)
> >
> > On Tue, Mar 10, 2015 at 6:47 PM, Kevin Burton <burton@spinn3r.com>
> wrote:
> >
> > > I’m still trying to track down some issues with ActiveMQ …
> > >
> > > One is that I have 5 ActiveMQ servers now, and each one has about 3000
> > > messages pending.  So 15000 messages in queues.
> > >
> > > These are non-persistent queues, plenty of memory and plenty of CPU,
> but
> > > the workers are just blocked waiting to receive work.
> > >
> > > I had a hypothesis that this could be slow workers, but after tuning
> some
> > > things I no longer receive any errors about slow workers.
> > >
> > > Restarting the daemons doesn’t fix things either.  Anything else it
> could
> > > be?  I’m a bit stumped unfortunately.
> > >
> > > --
> > >
> > > Founder/CEO Spinn3r.com
> > > Location: *San Francisco, CA*
> > > blog: http://burtonator.wordpress.com
> > > … or check out my Google+ profile
> > > <https://plus.google.com/102718274791889610666/posts>
> > > <http://spinn3r.com>
> > >
> >
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message