activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Broker refusing to send messages to only certain queues?
Date Fri, 17 Apr 2015 03:34:27 GMT
If that was happening you'd see the DestinationViewMBean's ExpiredCount
increasing in the JMX counters, but only if the expiration was happening on
the broker; as far as I could tell, there's no stat that captures when
messages expire while they're in the client's prefetch buffer.  (But that's
just from empirical observation, not code inspection, so I could be wrong
about that.)

If you're not seeing DLQ buildup, then that rules out the possibility of
messages aging off on the broker (as long as you haven't disabled DLQing of
expired messages via the processExpired property on
sharedDeadLetterStrategy).  I'm not sure if it rules out the possibility of
them aging off on the client; I'm not sure if messages get DLQ'ed in that
situation or simply discarded silently.  We don't use the DLQ at all, so I
don't have any empirical observations to work from, but maybe someone else
on the list would know that answer...  But even if it's possible, it
certainly sounds like a bit of a long shot for explaining what you're
seeing.

While this problem is going on, what are you seeing in JMX?  Which stats
are increasing, which ones aren't, which ones are going up slower than the
message count, etc.?

Tim

On Thu, Apr 16, 2015 at 2:10 PM, Kevin Burton <burton@spinn3r.com> wrote:

> actually, I not’ think this could be the issue.
>
> The if I was getting redeliveries, and NO work was being done, I’d see
> messages in the dead letter queue - which I don’t see.
>
>
>
> On Thu, Apr 16, 2015 at 9:55 AM, Kevin Burton <burton@spinn3r.com> wrote:
>
> > Ah..  That’s an interesting hypothesis.  So in this situation the client
> > would just never even see the message and would just keep being rejected.
> > Is there any way to test this hypothesis.? Any stats?
> >
> > I’ll look at JMX but I didn’t see any that might stand out.
> >
> > On Thu, Apr 16, 2015 at 5:51 AM, Tim Bain <tbain@alumni.duke.edu> wrote:
> >
> >> Is there any chance the messages are old enough that they're expired by
> >> the
> >> time the client gets to them?  Maybe the broker is running slowly enough
> >> that it doesn't try to dispatch each message until after it's expired,
> >> resulting in no messages getting *consumed* rather than no messages
> >> getting
> >> dispatched...
> >> On Apr 15, 2015 9:32 PM, "Kevin Burton" <burton@spinn3r.com> wrote:
> >>
> >> > And then if I immediately purge the queue, and add more messages to
> the
> >> > queues, they immediately work.  So it seems to be some bug related to
> >> > dispatch of the queue if it’s too large.
> >> >
> >> > On Wed, Apr 15, 2015 at 8:11 PM, Kevin Burton <burton@spinn3r.com>
> >> wrote:
> >> >
> >> > > It looks like this is a function of queue size.. if it grows large
> >> > enough,
> >> > > eventually, it seems, ActiveMQ stops dispatching messages to the
> >> > consumers.
> >> > >
> >> > > On Wed, Apr 15, 2015 at 6:55 PM, Kevin Burton <burton@spinn3r.com>
> >> > wrote:
> >> > >
> >> > >> I have a very strange bug I’m trying to track down.
> >> > >>
> >> > >> I have a very loaded ActiveMQ box (8 of them actually)… for
the
> most
> >> > part
> >> > >> it’s working very well.
> >> > >>
> >> > >> However, I recently noticed that some of our queues aren’t actually
> >> > doing
> >> > >> any work.  Other queues are performing just fine.  In fact, the
> vast
> >> > >> majority of them are performing fine.
> >> > >>
> >> > >> It’s just a few of them each with the same name.
> >> > >>
> >> > >> I’ve tried restarting both of my apps that consume my message,
only
> >> to
> >> > >> find the exact same behavior.
> >> > >>
> >> > >> The code is basically just not reading new messages sent to these
> >> given
> >> > >> queues.
> >> > >>
> >> > >> I can’t restart the broker because we’re using the memory
store for
> >> > >> performance reasons.
> >> > >>
> >> > >> I can browse them manually … the messages are there.  JMX reports
> >> > >> thousands of messages waiting to be executed.
> >> > >>
> >> > >> One theory I have is maybe the ActiveMQ connection isn’t properly
> >> > handing
> >> > >> messages to all my consumers.  The box is rather loaded.  I’m
also
> >> > calling
> >> > >> receiveNoWait.. to read these queues.  So perhaps some buffer
is
> >> being
> >> > >> filled up with other queues or some of my connections aren’t
being
> >> > serviced?
> >> > >>
> >> > >> Thoughts?
> >> > >>
> >> > >> --
> >> > >>
> >> > >> Founder/CEO Spinn3r.com
> >> > >> Location: *San Francisco, CA*
> >> > >> blog: http://burtonator.wordpress.com
> >> > >> … or check out my Google+ profile
> >> > >> <https://plus.google.com/102718274791889610666/posts>
> >> > >> <http://spinn3r.com>
> >> > >>
> >> > >>
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Founder/CEO Spinn3r.com
> >> > > Location: *San Francisco, CA*
> >> > > blog: http://burtonator.wordpress.com
> >> > > … or check out my Google+ profile
> >> > > <https://plus.google.com/102718274791889610666/posts>
> >> > > <http://spinn3r.com>
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> >
> >> > Founder/CEO Spinn3r.com
> >> > Location: *San Francisco, CA*
> >> > blog: http://burtonator.wordpress.com
> >> > … or check out my Google+ profile
> >> > <https://plus.google.com/102718274791889610666/posts>
> >> > <http://spinn3r.com>
> >> >
> >>
> >
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
> >
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message