activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: User Memory Issues
Date Tue, 25 Aug 2015 12:29:00 GMT
Fair enough, that was my first thought, but I also know that sometimes you
have to figure out a band-aid till the upgrade comes through.  But if
you're good just doing the upgrade, that sounds great to me!
On Aug 24, 2015 10:50 PM, "Daniel Israel" <disrael@liveops.com> wrote:

> Hi Tim,
>
> I appreciate your efforts, but it really doesn't make sense for me (and
> you) to try to debug the issue on an environment that's 7+ years old.  It's
> really on me to get the production system updated, I'm working on
> development environment now, which is where my questions come from.  Don't
> spend a lot of time trying to figure out what happened in version 5.2, it's
> totally appropriate that I should be on the hook to upgrade to a modern
> version.
>
> .... And thanks for the confirmation.  :)
>
>
>
>
>
> On 8/24/15, 9:18 PM, "tbain98@gmail.com on behalf of Tim Bain" <
> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>
> >Yes, as long as that's on the subscription, not the topic itself.  Caveat:
> >I haven't actually looked at JMX in 5.3.2, so I'm extrapolating from what
> >I've seen in later versions, but I think that should be right.
> >On Aug 24, 2015 1:29 PM, "Daniel Israel" <disrael@liveops.com> wrote:
> >
> >>
> >>
> >>
> >>
> >>
> >> On 8/21/15, 5:28 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >>
> >> >You can tell if a consumer is slower than the rest via JMX.  Look at
> each
> >> >subscription for the topic and see if any of them has lots of pending
> >> >messages while the rest do not.
> >>
> >> Is that the PendingQueueSize on the JMX console?
> >>
> >> Thanks.
> >>
> >>
> >> >If you discover that to be the case,
> >> >configure a slow consumer abortion strategy to kick them off and throw
> >> away
> >> >their pending messages.  (Note that this only works for topics, and
> only
> >> >for non-durable subscriptions, but I think both of those apply to your
> >> >scenario.)
> >> >On Aug 21, 2015 12:30 AM, "Daniel Israel" <disrael@liveops.com> wrote:
> >> >
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 8/20/15, 9:56 PM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >>
> >> >> >The broker can't discard the message till all subscribers have
> consumed
> >> >> it,
> >> >>
> >> >> This is good information, thanks!  I am wondering if there is a way
> to
> >> >> detect this?
> >> >>
> >> >> >so a single slow consumer will result in increased memory/store
> usage
> >> and
> >> >> >eventually PFC.  You could set up a slow consumer abortion strategy
> if
> >> >> >you're worried about that possibility.
> >> >> >On Aug 20, 2015 10:13 AM, "Daniel Israel" <disrael@liveops.com>
> wrote:
> >> >> >
> >> >> >> Oooh, that's a good idea to add another subber.  I may be
able to
> try
> >> >> that.
> >> >> >>
> >> >> >> Also, could something on the client side cause a backup? 
Or will
> the
> >> >> >> broker just fall through if a client hangs?
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On 8/20/15, 7:18 AM, "tbain98@gmail.com on behalf of Tim Bain"
<
> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >> >>
> >> >> >> >Since this is happening in production, I assume it's not
> acceptable
> >> to
> >> >> >> >attach a debugger and set a breakpoint on the line where
the
> >> exception
> >> >> is
> >> >> >> >thrown.  If you could, that would let you step into the
> >> >> >> >MemoryUsage.isFull() call and see what's going on, but
it'll hang
> >> the
> >> >> >> >broker when it happens and I assume that won't fly.
> >> >> >> >
> >> >> >> >You could probably figure out the size of the messages
just by
> >> >> creating a
> >> >> >> >test app that just logs the size of each message it receives
and
> >> >> having it
> >> >> >> >become another subscriber on the topic.  It won't tell
you the
> rate
> >> at
> >> >> >> >which your consumers consume, but you'll know the input
rates at
> >> least.
> >> >> >> >
> >> >> >> >If you're worried that your topics are filling up because
your
> >> >> consumers
> >> >> >> >are falling behind, you could force them to disconnect
(and throw
> >> away
> >> >> >> >their pending messages) if they get too far behind.
> >> >> >> >http://activemq.apache.org/slow-consumer-handling.html
 Just
> make
> >> sure
> >> >> >> the
> >> >> >> >limit you set is smaller than the PFC per-destination
limit, or
> >> you'll
> >> >> >> >never hit it.
> >> >> >> >
> >> >> >> >Tim
> >> >> >> >
> >> >> >> >On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <
> >> disrael@liveops.com>
> >> >> >> wrote:
> >> >> >> >
> >> >> >> >>
> >> >> >> >> Forgot to attach.  This is log message:
> >> >> >> >>
> >> >> >> >> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
> >> >> >> >>     - Async error occurred:
> javax.jms.ResourceAllocationException:
> >> >> Usage
> >> >> >> >> Manager memory limit reached
> >> >> >> >> javax.jms.ResourceAllocationException: Usage Manager
memory
> limit
> >> >> >> reached
> >> >> >> >>     at
> >> org.apache.activemq.broker.region.Topic.send(Topic.java:293)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> >> >> >> >>     at
> >> >> >>
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> >> >> >> >>     at java.lang.Thread.run(Thread.java:662)
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf
of Tim
> Bain" <
> >> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu>
wrote:
> >> >> >> >>
> >> >> >> >> >Hmm, the error messages I'm used to seeing (from
5.8.0 and
> >> 5.10.0)
> >> >> look
> >> >> >> >> >like the one in this page (
> >> >> >> >> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/
> ),
> >> >> which
> >> >> >> >> give
> >> >> >> >> >lots of information about what limit is being
hit.  I guess
> that
> >> >> >> detailed
> >> >> >> >> >info must have been added in a version after
the one you're
> >> using.
> >> >> >> What
> >> >> >> >> >version is that, anyway?
> >> >> >> >> >
> >> >> >> >> >Can you post the full log message?
> >> >> >> >> >
> >> >> >> >> >Have you explored the JMX tree to see if there
is any info to
> >> tell
> >> >> you
> >> >> >> how
> >> >> >> >> >full your topics are?  Obviously what information
is available
> >> will
> >> >> >> vary
> >> >> >> >> >based on what version you're running, so that
might not help,
> but
> >> >> you
> >> >> >> >> >should definitely check.
> >> >> >> >> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <
> disrael@liveops.com>
> >> >> wrote:
> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >> Hi Tim, thanks for response.
> >> >> >> >> >>
> >> >> >> >> >> Flow control is enabled, and it's configured
to fail if out
> of
> >> >> >> memory.
> >> >> >> >> As
> >> >> >> >> >> noted below, the log lines in this version
don't tell us
> which
> >> >> limit
> >> >> >> >> we're
> >> >> >> >> >> exceeding, so we're running half blind :(.
 Knowing average
> >> topic
> >> >> >> size
> >> >> >> >> >> would be helpful, but having individual
topic size would be
> >> good.
> >> >> >> Right
> >> >> >> >> >> now, looking at the producer side to see
if there is some
> way
> >> to
> >> >> >> track.
> >> >> >> >> >>
> >> >> >> >> >> Raised the topic limit to 20mb and still
had an issue.  It's
> >> very
> >> >> hit
> >> >> >> >> and
> >> >> >> >> >> miss.  I can run for a week without issue,
then when I get
> hit
> >> >> hard,
> >> >> >> it
> >> >> >> >> >> falls over.
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com
on behalf of Tim
> >> Bain" <
> >> >> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu>
> wrote:
> >> >> >> >> >>
> >> >> >> >> >> >Later versions give a few addition stats
(such as average
> >> message
> >> >> >> size)
> >> >> >> >> >> via
> >> >> >> >> >> >JMX, but that won't help you till that
upgrade in
> production
> >> is
> >> >> >> >> complete.
> >> >> >> >> >> >
> >> >> >> >> >> >Do you have producer flow control enabled?
 The error
> you're
> >> >> getting
> >> >> >> >> >> >doesn't match what I remember it being
the last time I hit
> >> it, so
> >> >> >> I'm
> >> >> >> >> >> >assuming you don't.  PFC gives log lines
that at least tell
> >> you
> >> >> >> exactly
> >> >> >> >> >> >which limit you ran into, plus it'll
avoid losing any
> messages
> >> >> (but
> >> >> >> >> it'll
> >> >> >> >> >> >"hang" producers till messages are consumed),
so you could
> >> enable
> >> >> >> it to
> >> >> >> >> >> >better understand what's going on.
> >> >> >> >> >> >
> >> >> >> >> >> >Tim
> >> >> >> >> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel"
<
> disrael@liveops.com
> >> >
> >> >> >> wrote:
> >> >> >> >> >> >
> >> >> >> >> >> >>
> >> >> >> >> >> >> Hello All,
> >> >> >> >> >> >>
> >> >> >> >> >> >> I am tracking down a memory issue
in AMQ.  I am time to
> time
> >> >> >> seeing
> >> >> >> >> >> >> exceptions like this:
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >> Async error occurred:
> javax.jms.ResourceAllocationException:
> >> >> Usage
> >> >> >> >> >> Manager
> >> >> >> >> >> >> memory limit reached
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >> I can't tell if this is because
I am exceeding the
> >> configured
> >> >> >> amount
> >> >> >> >> of
> >> >> >> >> >> >> memory in SystemUsage, or if I
am exceeding the amount of
> >> >> memory
> >> >> >> >> >> configured
> >> >> >> >> >> >> per topic.
> >> >> >> >> >> >>
> >> >> >> >> >> >> I am using only Topics right now,
and I had the memory
> limit
> >> >> set
> >> >> >> to
> >> >> >> >> >> 10mb.
> >> >> >> >> >> >> The error doesn't direct me in
any direction for this.
> I am
> >> >> >> using an
> >> >> >> >> >> old
> >> >> >> >> >> >> version of AMQ (First step was
to request upgrade to
> latest
> >> >> >> version.
> >> >> >> >> >> It's
> >> >> >> >> >> >> in the works, but it might be a
week or two before it's
> >> >> completed
> >> >> >> in
> >> >> >> >> >> >> production) and I see changes in
the source that give
> more
> >> >> details
> >> >> >> >> when
> >> >> >> >> >> >> throwing this exception.  Is there
some historical
> record or
> >> >> log
> >> >> >> of
> >> >> >> >> >> >> Topics?  What I'd really like is
to be able to see how
> often
> >> >> each
> >> >> >> >> Topic
> >> >> >> >> >> >> gets and distributes a message
and how big that message
> was.
> >> >> The
> >> >> >> >> >> dashboard
> >> >> >> >> >> >> and Jconsole give me some information,
but because Topics
> >> are
> >> >> >> >> delivered
> >> >> >> >> >> >> then released, I don't have any
information beyond how
> many
> >> >> were
> >> >> >> >> >> enqueued
> >> >> >> >> >> >> and delivered.
> >> >> >> >> >> >>
> >> >> >> >> >> >> Is there any such animal available
that would help me
> with
> >> >> this?
> >> >> >> Or
> >> >> >> >> >> >> suggestions how to approach?  Any
help is appreciated.
> >> Thanks.
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message