activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: I can finally somewhat duplicate the bad bug I’m seeing with ActiveMQ not dispatching messages.
Date Fri, 24 Apr 2015 23:21:49 GMT
Yes.. I think what the work around would probably do is revert to “softer”
priority scheduling when it kicks in because the fall through message
consumer with no selector will work.

I will still try to get a fix implemented of course but I have a release i
need to get out the door :)

On Fri, Apr 24, 2015 at 4:14 PM, Tim Bain <tbain@alumni.duke.edu> wrote:

> It might be a good fail-safe, but hopefully you'll find something as you
> step through with the debugger that will highlight what the problem
> actually is (so you don't need to use the no-selector workaround).
> Workarounds are great, but fixes are better.  :)  But it might be a way to
> at least get production working again so you can solve this at a more
> reasonable pace rather than spending 15-hour days on it.
>
> Tim
>
> On Fri, Apr 24, 2015 at 5:04 PM, Kevin Burton <burton@spinn3r.com> wrote:
>
> > Sounds like a good idea.  I just pushed with maxPageSize=5000
> >
> > Interesting that my backlogged queues seems to be locked up at about 5000
> > messages.
> >
> > … I’m wonder if those are messages that aren’t matching selectors?
> >
> > I could have a plan B / fall through that accepts messages without a
> > selector.. it would be a hack and would give me soft priority but this
> > would get me closer to my goal.
> >
> > throttle-queue:discovery-task:pending:192.0.78.13
> >                5,004 4
> > throttle-queue:discovery-task:pending:192.0.78.12
> >                5,002 4
> > throttle-queue:discovery-task:pending:74.125.141.132
> >                 5,000 0
> > throttle-queue:feed-task:pending:74.125.141.132
> >                4,997 0
> > throttle-queue:source-task:pending:69.171.237.20
> >                 4,062 0
> >
> > On Fri, Apr 24, 2015 at 3:40 PM, Tim Bain <tbain@alumni.duke.edu> wrote:
> >
> > > OK, so based on what you wrote, it sounds like the broker gets into a
> > state
> > > where message selectors don't work properly, since the same selector
> > > matches all messages (except ones with a null priority; have you
> > confirmed
> > > that both artemis_priority <= 9 and artemis_priority = null match their
> > > respective messages when you're in a good state, and that neither one
> > gets
> > > any messages when you're in a bad state?).
> > >
> > > I'd start the broker with the debug port enabled, pull up the
> > > selector-evaluation code and set a breakpoint in it (I'd start with
> > > SelectorWorker.run()), wait till you get into the bad state, and then
> > > attach a debugger and step through the code to see why messages aren't
> > > getting handed off to your consumer with the selector.
> > >
> > > Tim
> > >
> > > On Fri, Apr 24, 2015 at 4:27 PM, Kevin Burton <burton@spinn3r.com>
> > wrote:
> > >
> > > > > >
> > > > > > http://imgur.com/a/2myja
> > > > >
> > > > >
> > > > > What are the two screenshots; with and without the selector?  If
> > that's
> > > > > right, then clearly zero messages are matching the selector (which
> > > means
> > > > > this isn't a delivery problem, it's a selector problem), so
> hopefully
> > > > > pulling that thread will lead to a solution.
> > > > >
> > > > >
> > > > Actually, maybe it’s just a but related to message selectors?
> > > >
> > > > If I remove the selector, I have it print the messages it receives. I
> > > then
> > > > confirm that they are ALL have the right headers.  So from a
> > mathematical
> > > > perspective, they should match.
> > > >
> > > > Here’s an example.
> > > >
> > > > QueueDumperTask: Got message: using config: Config{sync=true,
> > > > selector='artemis_priority <= 9'}
> > > >
> > > > QueueDumperTask:
> > > > ID:util0048.wdc.sl.spinn3r.com-38049-1429893012757-6:1:1:1:11179
> > > >
> > > >   QueueDumperTask: artemis.link=
> > http://entertainment-2-all.blogspot.com
> > > >
> > > >   QueueDumperTask: artemis.throttleKey=74.125.141.132
> > > >
> > > >   QueueDumperTask: artemis.pending.timestamp=2015-04-24T16:46:30Z
> > > >
> > > >   QueueDumperTask: artemis.sequence=1429893990000012414
> > > >
> > > >   QueueDumperTask:
> > > >
> > artemis.pendingQueue=throttle-queue:discovery-task:pending:74.125.141.132
> > > >
> > > >   QueueDumperTask: artemis_priority=0
> > > >
> > > >   QueueDumperTask: artemis.throttleQueue=discovery-task
> > > >
> > > >   QueueDumperTask: artemis.delay=0
> > > >
> > > >   QueueDumperTask: JMSXDeliveryCount=3
> > > >
> > > > .. so it should match
> > > >
> > > > artemis_priority <= 9
> > > >
> > > > … so I think there are two issues here. I’m aware of the improper
> > > > scheduling issue do to maxPageSize… so I won’t get REAL priorities.
> > I’ll
> > > > get soft priorities.
> > > >
> > > > But that’s not the issue that I’m seeing.. What I’m seeing is that
> the
> > > > ENTIRE thing locks up.  Because at least ONE of my selectors should
> > work.
> > > >
> > > >
> > > > > > If I’m already an in-memory store, is there any problem just
> > setting
> > > > > > maxPageSize to a HUGE number?
> > > > >
> > > > >
> > > > > That's not code I've looked at, but I'd double-check the code to
> make
> > > > sure
> > > > > it's not going to do anything crazy like pre-allocate a huge
> > > > > array/ArrayList or anything like that...
> > > > >
> > > >
> > > > Yes. That’s what I was worried about.. I might trace it or just try
> it
> > in
> > > > production and see what memory usage looks like.
> > > >
> > > > --
> > > >
> > > > Founder/CEO Spinn3r.com
> > > > Location: *San Francisco, CA*
> > > > blog: http://burtonator.wordpress.com
> > > > … or check out my Google+ profile
> > > > <https://plus.google.com/102718274791889610666/posts>
> > > > <http://spinn3r.com>
> > > >
> > >
> >
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>



-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message