activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: Diagnosing long term ActiveMQ memory leaks?
Date Tue, 23 Dec 2014 23:15:34 GMT
The default GC which I think is still parallel.  But I should explicitly
set it.  I’m working on getting JMX monitors up so that I can track
ActiveMQ counters but also GC state.

I can’t get a reliable and short term failure to occur so right now I’ve
focused on mitigating the issue and easily rebuilding my queue when it
crashes.

On Tue, Dec 23, 2014 at 8:55 AM, Tim Bain <tbain@alumni.duke.edu> wrote:

> BTW, you never did answer the question about which GC strategy you're
> using, and it occurred to me that if you're using CMS, lots of full GCs
> that don't actually reclaim much memory after a long time being up is the
> classic failure scenario for CMS.  It happens when Old Gen gets fragmented,
> which in turn happens because CMS is a non-compacting GC strategy in Old
> Gen.  If you're using CMS and seeing continual full GCs, you should look at
> whether G1 GC would be better for your needs.
>
> On Mon, Dec 22, 2014 at 11:03 PM, Kevin Burton <burton@spinn3r.com> wrote:
>
> > Great feedback.  Thanks btw. I’m working on getting up better JMX
> monitors
> > so I can track memory here more aggressively.  Bumping up memory by 1.5G
> > temporarily fixed the problem. However, it seems correlated to the number
> > of connections.  So I suspect I’ll just hit this again in the next few
> > weeks.
> >
> > By that time I plan to have better JMX monitors in place to resolve this.
> >
> > On Sat, Dec 20, 2014 at 10:28 PM, Tim Bain <tbain@alumni.duke.edu>
> wrote:
> > >
> > > What JVM are you using, and what GC strategy with which options?  And
> for
> > > that matter, what broker version?
> > >
> > > With Hotspot 7u21 and G1GC while running a long-running performance
> > stress
> > > test I've observed that Old Gen use increases over time (despite the
> fact
> > > that G1GC is supposed to collect Old Gen during its normal collection
> > > operations), and GCs against Old Gen happen semi-continually after Old
> > Gen
> > > hits a certain memory threshold.  However, unlike what you're
> observing,
> > 1)
> > > the GCs I saw were Old Gen GCs but not full GCs (G1 allows GCing Old
> Gen
> > > during incremental GCs), 2) the broker remains responsive with
> reasonable
> > > pause times close to my target, and 3) once Old Gen hits the 90%
> > threshold
> > > that forces a full GC, that full GC is able to successfully collect
> > nearly
> > > all of the Old Gen memory.  My conclusion from that was that although
> > > objects were being promoted to Old Gen (and I tried unsuccessfully to
> > > prevent that from occurring, see
> > >
> > >
> >
> http://activemq.2283324.n4.nabble.com/Potential-Bug-in-Master-Slave-with-Replicated-LevelDB-Store-td4686450.html
> > > ),
> > > nearly all of them were unreachable by the time a full GC actually
> > > occurred.
> > >
> > > So if you're seeing continual full GCs (not just Old Gen GCs if you're
> > > using G1) that don't actually free any Old Gen memory, then what you're
> > > seeing different behavior than I saw, and it means that the objects in
> > Old
> > > Gen are still reachable.  One possible reason for that would be
> messages
> > > still being held in destinations waiting to be consumed; look for
> queues
> > > without consumers (especially DLQs), as well as durable subscribers
> that
> > > are offline.  If you're certain that's not the case, maybe you can post
> > > some of the results of analyzing the heap snapshot so that people who
> > know
> > > the codebase better could see if anything jumps out?
> > >
> > > On Sat, Dec 20, 2014 at 1:51 PM, Kevin Burton <burton@spinn3r.com>
> > wrote:
> > >
> > > > I’m trying to diagnose a long term memory leak with ActiveMQ.
> > > >
> > > > Basically , my app runs fine for about a week or so, then goes to
> 100%
> > > CPU
> > > > doing continually full GCs back to back.
> > > >
> > > > No work is done during that period.
> > > >
> > > > I have a large number of sessions to the AMQ box, but things are fine
> > on
> > > > startup.
> > > >
> > > > It’s entirely possible that y app isn’t releasing resources, but I”m
> > > trying
> > > > to figure out the best way to track that down.
> > > >
> > > > I’m using org.apache.activemq.UseDedicatedTaskRunner=false so that
> > thread
> > > > pools are used.  Which apparently can cause a bit of wasted memory.
> > > >
> > > > I have a heap snapshot.  I loaded that into the Eclipse Memory
> Analyzer
> > > and
> > > > didn’t see any obvious candidates but of course I’m not an expert
on
> > the
> > > > ActiveMQ code base.
> > > >
> > > > Are there any solid JMX counters I can track during this process?
> > Number
> > > > of sessions? etc.
> > > >
> > > > --
> > > >
> > > > Founder/CEO Spinn3r.com
> > > > Location: *San Francisco, CA*
> > > > blog: http://burtonator.wordpress.com
> > > > … or check out my Google+ profile
> > > > <https://plus.google.com/102718274791889610666/posts>
> > > > <http://spinn3r.com>
> > > >
> > >
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>



-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message