activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.
Date Fri, 24 Apr 2015 23:14:37 GMT
It might be a good fail-safe, but hopefully you'll find something as you
step through with the debugger that will highlight what the problem
actually is (so you don't need to use the no-selector workaround).
Workarounds are great, but fixes are better.  :)  But it might be a way to
at least get production working again so you can solve this at a more
reasonable pace rather than spending 15-hour days on it.

Tim

On Fri, Apr 24, 2015 at 5:04 PM, Kevin Burton <burton@spinn3r.com> wrote:

> Sounds like a good idea.  I just pushed with maxPageSize=5000
>
> Interesting that my backlogged queues seems to be locked up at about 5000
> messages.
>
> … I’m wonder if those are messages that aren’t matching selectors?
>
> I could have a plan B / fall through that accepts messages without a
> selector.. it would be a hack and would give me soft priority but this
> would get me closer to my goal.
>
> throttle-queue:discovery-task:pending:192.0.78.13
>                5,004 4
> throttle-queue:discovery-task:pending:192.0.78.12
>                5,002 4
> throttle-queue:discovery-task:pending:74.125.141.132
>                 5,000 0
> throttle-queue:feed-task:pending:74.125.141.132
>                4,997 0
> throttle-queue:source-task:pending:69.171.237.20
>                 4,062 0
>
> On Fri, Apr 24, 2015 at 3:40 PM, Tim Bain <tbain@alumni.duke.edu> wrote:
>
> > OK, so based on what you wrote, it sounds like the broker gets into a
> state
> > where message selectors don't work properly, since the same selector
> > matches all messages (except ones with a null priority; have you
> confirmed
> > that both artemis_priority <= 9 and artemis_priority = null match their
> > respective messages when you're in a good state, and that neither one
> gets
> > any messages when you're in a bad state?).
> >
> > I'd start the broker with the debug port enabled, pull up the
> > selector-evaluation code and set a breakpoint in it (I'd start with
> > SelectorWorker.run()), wait till you get into the bad state, and then
> > attach a debugger and step through the code to see why messages aren't
> > getting handed off to your consumer with the selector.
> >
> > Tim
> >
> > On Fri, Apr 24, 2015 at 4:27 PM, Kevin Burton <burton@spinn3r.com>
> wrote:
> >
> > > > >
> > > > > http://imgur.com/a/2myja
> > > >
> > > >
> > > > What are the two screenshots; with and without the selector?  If
> that's
> > > > right, then clearly zero messages are matching the selector (which
> > means
> > > > this isn't a delivery problem, it's a selector problem), so hopefully
> > > > pulling that thread will lead to a solution.
> > > >
> > > >
> > > Actually, maybe it’s just a but related to message selectors?
> > >
> > > If I remove the selector, I have it print the messages it receives. I
> > then
> > > confirm that they are ALL have the right headers.  So from a
> mathematical
> > > perspective, they should match.
> > >
> > > Here’s an example.
> > >
> > > QueueDumperTask: Got message: using config: Config{sync=true,
> > > selector='artemis_priority <= 9'}
> > >
> > > QueueDumperTask:
> > > ID:util0048.wdc.sl.spinn3r.com-38049-1429893012757-6:1:1:1:11179
> > >
> > >   QueueDumperTask: artemis.link=
> http://entertainment-2-all.blogspot.com
> > >
> > >   QueueDumperTask: artemis.throttleKey=74.125.141.132
> > >
> > >   QueueDumperTask: artemis.pending.timestamp=2015-04-24T16:46:30Z
> > >
> > >   QueueDumperTask: artemis.sequence=1429893990000012414
> > >
> > >   QueueDumperTask:
> > >
> artemis.pendingQueue=throttle-queue:discovery-task:pending:74.125.141.132
> > >
> > >   QueueDumperTask: artemis_priority=0
> > >
> > >   QueueDumperTask: artemis.throttleQueue=discovery-task
> > >
> > >   QueueDumperTask: artemis.delay=0
> > >
> > >   QueueDumperTask: JMSXDeliveryCount=3
> > >
> > > .. so it should match
> > >
> > > artemis_priority <= 9
> > >
> > > … so I think there are two issues here. I’m aware of the improper
> > > scheduling issue do to maxPageSize… so I won’t get REAL priorities.
> I’ll
> > > get soft priorities.
> > >
> > > But that’s not the issue that I’m seeing.. What I’m seeing is that the
> > > ENTIRE thing locks up.  Because at least ONE of my selectors should
> work.
> > >
> > >
> > > > > If I’m already an in-memory store, is there any problem just
> setting
> > > > > maxPageSize to a HUGE number?
> > > >
> > > >
> > > > That's not code I've looked at, but I'd double-check the code to make
> > > sure
> > > > it's not going to do anything crazy like pre-allocate a huge
> > > > array/ArrayList or anything like that...
> > > >
> > >
> > > Yes. That’s what I was worried about.. I might trace it or just try it
> in
> > > production and see what memory usage looks like.
> > >
> > > --
> > >
> > > Founder/CEO Spinn3r.com
> > > Location: *San Francisco, CA*
> > > blog: http://burtonator.wordpress.com
> > > … or check out my Google+ profile
> > > <https://plus.google.com/102718274791889610666/posts>
> > > <http://spinn3r.com>
> > >
> >
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message