activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Diagnosing long term ActiveMQ memory leaks?
Date Tue, 23 Dec 2014 16:55:15 GMT
BTW, you never did answer the question about which GC strategy you're
using, and it occurred to me that if you're using CMS, lots of full GCs
that don't actually reclaim much memory after a long time being up is the
classic failure scenario for CMS.  It happens when Old Gen gets fragmented,
which in turn happens because CMS is a non-compacting GC strategy in Old
Gen.  If you're using CMS and seeing continual full GCs, you should look at
whether G1 GC would be better for your needs.

On Mon, Dec 22, 2014 at 11:03 PM, Kevin Burton <burton@spinn3r.com> wrote:

> Great feedback.  Thanks btw. I’m working on getting up better JMX monitors
> so I can track memory here more aggressively.  Bumping up memory by 1.5G
> temporarily fixed the problem. However, it seems correlated to the number
> of connections.  So I suspect I’ll just hit this again in the next few
> weeks.
>
> By that time I plan to have better JMX monitors in place to resolve this.
>
> On Sat, Dec 20, 2014 at 10:28 PM, Tim Bain <tbain@alumni.duke.edu> wrote:
> >
> > What JVM are you using, and what GC strategy with which options?  And for
> > that matter, what broker version?
> >
> > With Hotspot 7u21 and G1GC while running a long-running performance
> stress
> > test I've observed that Old Gen use increases over time (despite the fact
> > that G1GC is supposed to collect Old Gen during its normal collection
> > operations), and GCs against Old Gen happen semi-continually after Old
> Gen
> > hits a certain memory threshold.  However, unlike what you're observing,
> 1)
> > the GCs I saw were Old Gen GCs but not full GCs (G1 allows GCing Old Gen
> > during incremental GCs), 2) the broker remains responsive with reasonable
> > pause times close to my target, and 3) once Old Gen hits the 90%
> threshold
> > that forces a full GC, that full GC is able to successfully collect
> nearly
> > all of the Old Gen memory.  My conclusion from that was that although
> > objects were being promoted to Old Gen (and I tried unsuccessfully to
> > prevent that from occurring, see
> >
> >
> http://activemq.2283324.n4.nabble.com/Potential-Bug-in-Master-Slave-with-Replicated-LevelDB-Store-td4686450.html
> > ),
> > nearly all of them were unreachable by the time a full GC actually
> > occurred.
> >
> > So if you're seeing continual full GCs (not just Old Gen GCs if you're
> > using G1) that don't actually free any Old Gen memory, then what you're
> > seeing different behavior than I saw, and it means that the objects in
> Old
> > Gen are still reachable.  One possible reason for that would be messages
> > still being held in destinations waiting to be consumed; look for queues
> > without consumers (especially DLQs), as well as durable subscribers that
> > are offline.  If you're certain that's not the case, maybe you can post
> > some of the results of analyzing the heap snapshot so that people who
> know
> > the codebase better could see if anything jumps out?
> >
> > On Sat, Dec 20, 2014 at 1:51 PM, Kevin Burton <burton@spinn3r.com>
> wrote:
> >
> > > I’m trying to diagnose a long term memory leak with ActiveMQ.
> > >
> > > Basically , my app runs fine for about a week or so, then goes to 100%
> > CPU
> > > doing continually full GCs back to back.
> > >
> > > No work is done during that period.
> > >
> > > I have a large number of sessions to the AMQ box, but things are fine
> on
> > > startup.
> > >
> > > It’s entirely possible that y app isn’t releasing resources, but I”m
> > trying
> > > to figure out the best way to track that down.
> > >
> > > I’m using org.apache.activemq.UseDedicatedTaskRunner=false so that
> thread
> > > pools are used.  Which apparently can cause a bit of wasted memory.
> > >
> > > I have a heap snapshot.  I loaded that into the Eclipse Memory Analyzer
> > and
> > > didn’t see any obvious candidates but of course I’m not an expert on
> the
> > > ActiveMQ code base.
> > >
> > > Are there any solid JMX counters I can track during this process?
> Number
> > > of sessions? etc.
> > >
> > > --
> > >
> > > Founder/CEO Spinn3r.com
> > > Location: *San Francisco, CA*
> > > blog: http://burtonator.wordpress.com
> > > … or check out my Google+ profile
> > > <https://plus.google.com/102718274791889610666/posts>
> > > <http://spinn3r.com>
> > >
> >
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message