activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Thousands of workers waiting to receive jobs but ActiveMQ isn’t sending them.
Date Wed, 11 Mar 2015 12:33:20 GMT
I'd definitely set breakpoints and step through with a debugger to try to
figure out what's going on.
On Mar 10, 2015 8:19 PM, "Kevin Burton" <burton@spinn3r.com> wrote:

> This is exceedingly bizarre.  Now ActiveMQ is refusing to deliver ANY
> messages to my workers.
>
> This is very bizarre, no code has changed.  Nothing.  It’s just refusing to
> give work.
>
> If I set the prefetch to 0 or 1, it does work for a few moments, then
> halts.
>
> 99% certain I’m committing all my messages.  As it would make sense that
> nothing could be processed after that of course.
>
> Kevin
>
>
> On Tue, Mar 10, 2015 at 6:53 PM, Tim Bain <tbain@alumni.duke.edu> wrote:
>
> > If you make a single consumer, you'll only get one message at a time by
> > default (so only one thread will be doing any work).  You'd have to use
> > client acknowledgement or selective acknowledgement to get more than one
> > message at a time.  I'd probably leave many consumers but tune down your
> > prefetch buffers to something relatively small to ensure that the
> workload
> > is evenly spread and you don't have some consumers with a large prefetch
> > buffer worth of backlog while others sit around idle.
> >
> > But if you're seeing the broker report pending messages, then that means
> > that having unbalanced workloads due to large prefetch buffers isn't your
> > problem...  Pending messages on the broker only occur when those messages
> > can't be dispatched to any consumer because all of their prefetch buffers
> > are full, which would mean that you don't have unbalanced workloads.
> > You'll get one or the other, not both.
> >
> > On Tue, Mar 10, 2015 at 7:42 PM, Kevin Burton <burton@spinn3r.com>
> wrote:
> >
> > > I’m actually wondering if this is my issues. I’m creating one session
> per
> > > thread.  So perhaps some of the threads have work to do, but they’re
> each
> > > prefetching a bunch of work when in reality a better strategy might the
> > to
> > > have one master listener and then dispatch messages to each thread.
> > >
> > > On Tue, Mar 10, 2015 at 6:37 PM, Kevin Burton <burton@spinn3r.com>
> > wrote:
> > >
> > > > OK.  That’s good to know.  I have a large number of connections so I
> > have
> > > > to look at each one.  I wonder if this could also be the issue.  AKA
> > too
> > > > many connections.
> > > >
> > > > On Tue, Mar 10, 2015 at 6:19 PM, Tim Bain <tbain@alumni.duke.edu>
> > wrote:
> > > >
> > > >> You should be able to confirm that the prefetch buffers are empty
by
> > > >> inspecting the JMX MBeans on the broker.  Look at the consumers for
> > the
> > > >> destination, and for each one look at its DispatchedQueueSize
> > attribute.
> > > >>
> > > >> Keep in mind that slow consumers are identified *ONLY* if you
> > configure
> > > >> one
> > > >> of the abort strategies.  If you didn't set that up, don't expect
> any
> > > slow
> > > >> consumer identification log lines.  And if you did, I've never seen
> a
> > > >> situation where a consumer went slow and a log line didn't happen
> > (using
> > > >> the SlowConsumerAbortStrategy; I haven't used
> > > SlowAckConsumerAbortStrategy
> > > >> and can't vouch for it); we get those log lines pretty frequently.
> So
> > > if
> > > >> you're not seeing broker-side log lines about consumers being
> > identified
> > > >> as
> > > >> slow and then aborted, I'd bet it's simply not happening.
> > > >>
> > > >> On Tue, Mar 10, 2015 at 7:10 PM, Kevin Burton <burton@spinn3r.com>
> > > wrote:
> > > >>
> > > >> > The broker.  I’ll assume the prefetch brokers are empty. I’m
> looking
> > > >> into
> > > >> > debugging that now but I don’t have tools to introspect.
> > > >> >
> > > >> > The broker has thousands of messages.
> > > >> >
> > > >> > I just confirmed that a restart DOES improve the situation.
> > > >> >
> > > >> > It’s possible that they’re being marked as slow consumers
but not
> > > >> *logged*
> > > >> > as such so I’m trying to use JMX to dump the sessions.
> > > >> >
> > > >> > On Tue, Mar 10, 2015 at 5:58 PM, Tim Bain <tbain@alumni.duke.edu>
> > > >> wrote:
> > > >> >
> > > >> > > Are the messages getting hung up in the broker or in the
client?
> > > (Do
> > > >> the
> > > >> > > consumers have empty or full prefetch buffers?)
> > > >> > >
> > > >> > > On Tue, Mar 10, 2015 at 6:47 PM, Kevin Burton <
> burton@spinn3r.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > > > I’m still trying to track down some issues with ActiveMQ
…
> > > >> > > >
> > > >> > > > One is that I have 5 ActiveMQ servers now, and each
one has
> > about
> > > >> 3000
> > > >> > > > messages pending.  So 15000 messages in queues.
> > > >> > > >
> > > >> > > > These are non-persistent queues, plenty of memory and
plenty
> of
> > > CPU,
> > > >> > but
> > > >> > > > the workers are just blocked waiting to receive work.
> > > >> > > >
> > > >> > > > I had a hypothesis that this could be slow workers,
but after
> > > tuning
> > > >> > some
> > > >> > > > things I no longer receive any errors about slow workers.
> > > >> > > >
> > > >> > > > Restarting the daemons doesn’t fix things either.
 Anything
> else
> > > it
> > > >> > could
> > > >> > > > be?  I’m a bit stumped unfortunately.
> > > >> > > >
> > > >> > > > --
> > > >> > > >
> > > >> > > > Founder/CEO Spinn3r.com
> > > >> > > > Location: *San Francisco, CA*
> > > >> > > > blog: http://burtonator.wordpress.com
> > > >> > > > … or check out my Google+ profile
> > > >> > > > <https://plus.google.com/102718274791889610666/posts>
> > > >> > > > <http://spinn3r.com>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > Founder/CEO Spinn3r.com
> > > >> > Location: *San Francisco, CA*
> > > >> > blog: http://burtonator.wordpress.com
> > > >> > … or check out my Google+ profile
> > > >> > <https://plus.google.com/102718274791889610666/posts>
> > > >> > <http://spinn3r.com>
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Founder/CEO Spinn3r.com
> > > > Location: *San Francisco, CA*
> > > > blog: http://burtonator.wordpress.com
> > > > … or check out my Google+ profile
> > > > <https://plus.google.com/102718274791889610666/posts>
> > > > <http://spinn3r.com>
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Founder/CEO Spinn3r.com
> > > Location: *San Francisco, CA*
> > > blog: http://burtonator.wordpress.com
> > > … or check out my Google+ profile
> > > <https://plus.google.com/102718274791889610666/posts>
> > > <http://spinn3r.com>
> > >
> >
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message