activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: consumer prefetch considered harmful ..evict from the prefetch buffer?
Date Tue, 09 Jun 2015 13:54:55 GMT
On Mon, Jun 8, 2015 at 9:58 AM, Kevin Burton <burton@spinn3r.com> wrote:

> >
> > I can see two potential problems that your description didn't draw a line
> > between:
> >
> >    1. With a large prefetch buffer, it's possible to have one thread
> have a
> >    large number of prefetched tasks and another have none, even if all
> > tasks
> >    take an average amount of time to complete.  No thread is slow per se,
> > but
> >    because the messages were prefetched lopsidedly, one thread sits idle
> > while
> >    the other churns through what's on its plate.
> >
>
>
> Yes.  Totally.  I think I mentioned that but maybe didn’t spell it out
> perfectly.
>
> This is one major edge case that needs to be addressed.
>
>
> >    2. With *any* prefetch buffer size, it's possible to have one message
> >    that takes forever to complete.  Any messages caught behind that one
> > slow
> >    message are stuck until it finishes.
> >
> >
> No.. that won’t happen because I have one consumer per thread so others are
> dispatched on the other consumers.  Even if prefetch is one.
>
> At least I have a test for this, and believe this to be the case and
> verified that my test works properly.
>
> But this you *may* be right if we’re just explaining it differently.  I use
> one thread per consumer, so as long as there’s a message in prefetch, then
> I’m good.
>

Prefetch buffers are per-consumer, not per-connection or per-thread or
per-anything-else.  A consumer that has messages other than the current one
prefetched and takes a long time to process the current message will
prevent any consumers from processing the messages it has prefetched.  I'm
not sure what your test is testing/showing, but I'm skeptical that it's
showing that a slow consumer with > 1 message prefetched allows other
consumers to process the messages it's not getting to.

the problem is, I think, that my high CPU is stalling out ActiveMQ and so I
> can’t stay prefetched.
>
>
> >  Which scenario are you worried about here?
> >
> > If the latter, the AbortSlowAckConsumerStrategy (
> >
> >
> http://timbish.blogspot.com/2013/07/coming-in-activemq-59-new-way-to-abort.html
> > ;
> > sadly the wiki doesn't detail this strategy and Tim's personal blog post
> is
> > the best documentation available) is intended to address exactly this:
> >
>
> Oh yes.. I think I remember reading this.
>
> yes.. the duplicate processing is far from ideal.
>
>
> > If the former, you're basically looking to enable work-stealing between
> > consumers, and I'm not aware of any existing capability to do that.  If
> you
> > wanted to implement it, you'd probably want to implement it as a sibling
> > class to AbortSlowAckConsumerStrategy where SlowAck is the trigger but
> > StealWork is the action rather than Abort.
>
>
>
> Yes.. but I think you detailed the reason why it’s not ideal - it requires
> a lot of work!
>

Absolutely, plus changes to the OpenWire protocol (and the need to maintain
backwards compatibility with OpenWire versions).  Which is why the typical
solution to this problem is to use a low prefetch buffer size (so you're
only stalling out a few messages) and just live with the delay on those
couple of messages.  But a better solution would be awesome, if you wanted
to work towards it.


> I'm a little skeptical that your worker threads could so thoroughly smother
> > the CPU that the thread doing the prefetching gets starved out
> >
>
> you underestimate the power of the dark side… :)
>
> We’re very high CPU load… with something like 250-500 threads per daemon.
>
> We’re CPU oriented so if there’s work to be done, and we’re not at 100%
> CPU, then we’re wasting compute resources.
>

Eeek.  Are you sure you're not wasting more resources switching contexts
than you're gaining by nominally keeping a thread on all cores at all
times?  (Some of that CPU time is being spent moving threads around rather
than running them.)


> > (particularly since I'd expect it to be primarily I/O-bound, so it's CPU
> > usage should be minimal), though I guess if you had as many worker
> threads
> > as cores you might be able to burn through all the prefetched messages
> > before the ActiveMQ thread gets rescheduled.  But I assume that your
> > workers are doing non-trivial amounts of work and are probably getting
> > context switched repeatedly during their processing, which I'd think
> would
> > give the ActiveMQ thread plenty of time to do what it needs to.  Unless
> 1)
> > you've set thread priorities to prioritize your workers over ActiveMQ, in
> > which case don't do that,
>
>
>
> Yes.  My thread should be minimum priority.  I would verify that but
> there’s a Linux bug that causes jstack to show the wrong value.
>
> Does anyone know what priority the ActiveMQ transport thread runs under?
> The above bug prevents me from (easily) figuring that out.
>

I don't know what priority it runs as, but could you figure it out by
attaching a debugger and setting a breakpoint and then examining the thread
in the Watch/Expressions windows?  You might even try raising that thread's
priority (via the debugger, initially) and seeing if it makes an
improvement.


> > 2) your worker threads are somehow holding onto a
> > lock that the ActiveMQ thread needs, which is possible but seems
> unlikely,
> > or 3) you've set up so many consumers (far more than you have cores) that
> > the 1/(N+1)th that the ActiveMQ thread gets is too little or too
> infrequent
> > to maintain responsiveness, in which case you need to scale back your
> > worker thread pool size (which I think means using fewer consumers per
> > process, based on what you've described).
> >
>
> Yes.  This is the case.  What I’m thinking to do is the reverse.  To use
> more connections and put then in a connection pull ..
>
> To right now if I have 1 connection, and 500 workers, then I have a 1:500
> ratio..  But if I bump that up to just 10… I’ll have 1:50.
>
> It think this is more realistic and would mean more CPU time to keep the
> prefetch buffer warm.  If I sized it as 1:1 (which would be wasteful of
> resources I think) then I think the problem would effectively be solved
> (but waste memory to solve it).
>
> But maybe around 1:10 or 1:20 it would be resolved.
>
> I need to verify that ActiveMQ allocates more thread per connection but I’m
> pretty sure it does.
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message