activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: A proposal to rewrite purgeInactiveDestinations locking to prevent queue GC lockups.
Date Sun, 22 Feb 2015 23:03:43 GMT
On Sun, Feb 22, 2015 at 2:43 PM, Tim Bain <tbain@alumni.duke.edu> wrote:

> I like the more-granular lock idea; that's better than holding the lock
> with brief unlocking periods as I proposed in the email I sent just now to
> your other thread.  In practice you'll need to do the refactoring I
> proposed in order to implement the algorithm you've described here, only
> you'll add a lookup for figuring out the lock for this queue.
>
>
I just responded to that email but I agree with you.


> I think this is much better than relying on the low max number of queues
> and trying to do just a few at a time.  Do you know whether what's slow is
> the call to canGC() or the call to removeDestination()?  If it's canGC(),
> the do-it-often-but-only-for-a-few approach will kill you performance-wise
> because you'll spend large amounts of time holding the lock while calling
> canGC on active destinations as you look for any that are inactive.  So for
> multiple reasons, don't do that.
>
>
I don’t know yet..  I suspect removeDestination because I always see it
there in the stack trace and I also suspect that 99.% of this is LevelDB.
It’s just that this global lock blocking everything means that ActiveMQ is
dead in the water while GC is happening.

What I was going to do was look at installing a snapshot and then starting
to work with that snapshot for a while.  I just built it with a tar.gz so I
can use that to push changes into production easily.

I wanted to put a timer / stopwatch around the ENTIRE gc loop to measure
the total time it takes (we really need to trace the duration of that) and
also can add one around canGC and removeDestination.


> Definitely submit an enhancement request in JIRA, even if you're not sure
> you'll need it.  Let's not lose track of the idea now that you've put all
> this time into digging into it.
>

Sounds like a plan. I have a test project that I can use now to reference
and I’ll create a JIRA for this.  I have to upgrade to 5.11 anyway so if I
can just take the current branch, build it, then create my .deb from it
then I can easily push test code into production to verify that we’ve
resolved this bug.

Kevin

-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message