activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fred Moore <fred.moor...@gmail.com>
Subject Re: OOM with high KahaDB index time
Date Mon, 25 Jan 2010 14:52:14 GMT
Hi Joe,

can you post your whole activemq.xml for this "good enough" test on 5.3?

...I'm still missing something and getting OOMs.
TIA,
F.


On Fri, Jan 22, 2010 at 9:07 PM, Joe Fernandez <
joe.fernandez@ttmsolutions.com> wrote:

>
> I ran my 5.3 test with the following
>
>    <systemUsage>
>            <systemUsage sendFailIfNoSpace="true">
>                <memoryUsage>
>                    <memoryUsage limit="100mb"/>
>                </memoryUsage>
>                <storeUsage>
>                    <storeUsage limit="500 mb" name="foo"/>
>                 </storeUsage>
>                <tempUsage>
>                    <tempUsage limit="100mb"/>
>                </tempUsage>
>            </systemUsage>
>        </systemUsage>
>
> ...and had my producer fill up the store; it tried to pump 400k messages
> each at 2k in size. Producer flow control was also enabled. The broker did
> not hurl any OOM exceptions, "just javax.jms.ResourceAllocationException:
> Usage Manager Store is Full. Stopping producer..." exceptions as expected.
> The producer got kicked off its connection.
>
> The only time I got OOMs was when I used this.
>
> <systemUsage>
>     <systemUsage sendFailIfNoSpace="true">
>     </systemUsage>
> </systemUsage>
>
> Joe
>
>
>
> Daniel Kluesing-2 wrote:
> >
> > So I'm not sure if that's what Rob is talking about being fixed in 5.4
> > (And I'll try the snapshot as soon as it's ready) but if I don't have the
> > sendFailIfNoSpace then my understanding is the producers send calls
> > block/wait/timeout - as opposed to fail - so it's more difficult to get
> > into a HA configuration. It's a minor point, not having an OOM is much
> > more important, but I definitely want the send calls to fail for the
> > producer if the broker ever does anything funny.
> >
> > Thanks for the feedback on the config, very helpful.
> >
> > -----Original Message-----
> > From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
> > Sent: Thursday, January 21, 2010 1:01 PM
> > To: users@activemq.apache.org
> > Subject: RE: OOM with high KahaDB index time
> >
> >
> > Just for grins, I threw your cfg file into our 5.3 testbed and sure
> > enough,
> > we got the OOMs; I pumped 200k messages with each being 2k in size. FWIW,
> > taking this out of the cfg file made things run a lot better.
> >
> >  <systemUsage>
> >      <systemUsage sendFailIfNoSpace="true">
> >      </systemUsage>
> > </systemUsage>
> >
> > With the above taken out of the cfg file, I was able to pump 400k
> messages
> > into the broker, no OOMs and memory utilization looked much better. I
> also
> > gave a fully-defined systemUsage a try and that also appeared to do the
> > trick.
> >
> >  <systemUsage>
> >             <systemUsage>
> >                 <memoryUsage>
> >                     <memoryUsage limit="100 mb"/>
> >                 </memoryUsage>
> >                 <storeUsage>
> >                     <storeUsage limit="1 gb" name="foo"/>
> >                 </storeUsage>
> >                 <tempUsage>
> >                     <tempUsage limit="100 mb"/>
> >                 </tempUsage>
> >             </systemUsage>
> >  </systemUsage>
> >
> > So may be worth giving it a whirl if you can't scoot over to the trunk
> and
> > ride Rob's patch.
> >
> > Joe
> >
> >
> > Daniel Kluesing-2 wrote:
> >>
> >> I tried the suggestion of going with the default cursor, but I still get
> >> OOM errors. I've included my full config file below, I think I'm running
> >> fairly vanilla/default.
> >>
> >> After about 350k persistent messages, the logs start to look like:
> >>
> >> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took
> >> 3118 ms
> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> >> 5118 ms
> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> >> 2736 ms
> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> >> 2945 ms
> >>  INFO | Slow KahaDB access: Journal append took: 33 ms, Index update
> took
> >> 2654 ms
> >>  INFO | Slow KahaDB access: Journal append took: 82 ms, Index update
> took
> >> 3174 ms
> >>  INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took
> >> 5891 ms
> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> >> 2906 ms
> >>  INFO | Slow KahaDB access: Journal append took: 60 ms, Index update
> took
> >> 7619 ms
> >> Exception in thread "InactivityMonitor WriteCheck"
> >> java.lang.OutOfMemoryError: Java heap space
> >>         at java.util.jar.Attributes.read(Attributes.java:377)
> >>         at java.util.jar.Manifest.read(Manifest.java:182)
> >>         at java.util.jar.Manifest.<init>(Manifest.java:52)
> >>         at
> >> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
> >>         at java.util.jar.JarFile.getManifest(JarFile.java:146)
> >>         at
> >> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
> >>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
> >>         at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> >>         at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> >>         at java.security.AccessController.doPrivileged(Native Method)
> >>         at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>         at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> >>         at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> >>         at
> >>
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
> >>         at
> >>
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
> >>         at
> >>
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> >>         at java.util.TimerThread.mainLoop(Timer.java:512)
> >>         at java.util.TimerThread.run(Timer.java:462)
> >>
> >> Config file:
> >>
> >> <beans
> >>   xmlns="http://www.springframework.org/schema/beans"
> >>   xmlns:amq="http://activemq.apache.org/schema/core"
> >>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >>   xsi:schemaLocation="http://www.springframework.org/schema/beans
> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> >>   http://activemq.apache.org/schema/core
> >> http://activemq.apache.org/schema/core/activemq-core.xsd">
> >>
> >>     <bean
> >>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> >>         <property name="locations">
> >>
> >> <value>file:${activemq.base}/conf/credentials.properties</value>
> >>         </property>
> >>     </bean>
> >>
> >>     <broker xmlns="http://activemq.apache.org/schema/core"
> >> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
> >>
> >>         <managementContext>
> >>             <managementContext createConnector="true"/>
> >>         </managementContext>
> >>
> >>         <persistenceAdapter>
> >>             <kahaDB directory="${activemq.base}/data/kahadb"/>
> >>         </persistenceAdapter>
> >>
> >>      <destinationPolicy>
> >>                      <policyMap>
> >>                              <policyEntries>
> >>                              <policyEntry queue="P>"
> producerFlowControl="true"
> >> memoryLimit="10mb"></policyEntry>
> >>                      </policyEntries>
> >>                      </policyMap>
> >>      </destinationPolicy>
> >>         <systemUsage>
> >>             <systemUsage sendFailIfNoSpace="true">
> >>             </systemUsage>
> >>         </systemUsage>
> >>         <transportConnectors>
> >>             <transportConnector name="openwire"
> >> uri="tcp://0.0.0.0:61616"/>
> >>         </transportConnectors>
> >>     </broker>
> >>     <import resource="jetty.xml"/>
> >> </beans>
> >>
> >> -----Original Message-----
> >> From: Rob Davies [mailto:rajdavies@gmail.com]
> >> Sent: Monday, January 18, 2010 10:42 PM
> >> To: users@activemq.apache.org
> >> Subject: Re: OOM with high KahaDB index time
> >>
> >>
> >> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
> >>
> >>> Hi,
> >>>
> >>> I'm running the 5.3 release as a standalone broker. In one case, a
> >>> producer is running without a consumer, producing small, persistent
> >>> messages, with the FileCursor pendingQueuePolicy (per
> >>> https://issues.apache.org/activemq/browse/AMQ-2512)
> >>>  option and flow control memoryLimit set to 100mb for the queue in
> >>> question. (Through a policy entry)
> >>>
> >>> As the queue grows above 300k messages, KahaDB indexing starts
> >>> climbing above 1 second. At around 350k messages, the indexing is
> >>> taking over 8 seconds. At this point, I start getting java out of
> >>> heap space errors in essentially random parts of the code. After a
> >>> while, the producers timeout with a channel inactive for too long
> >>> error, and the entire broker basically wedges itself. At this point,
> >>> consumers are generally unable to bind to the broker quitting with
> >>> timeout errors. When they can connect, consuming a single message
> >>> triggers an index re-build, which takes 2-8seconds. Turning on
> >>> verbose garbage collection, the jvm is collecting like mad but
> >>> reclaiming no space.
> >>>
> >>> If I restart the broker, it comes back up, I can consume the old
> >>> messages, and can handle another 350k messages until it wedges.
> >>>
> >>> I can reproduce under both default gc and incremental gc.
> >>>
> >>> Two questions:
> >>> - It seems like someone is holding onto a handle to the messages
> >>> after they have been persisted to disk - is this a known issue?
> >>> Should I open a JIRA for it? (Or is there another explanation?)
> >>>
> >>> - Is there any documentation about the internals of KahaDB - the
> >>> kind of indices etc? I'd like to get a better understanding of the
> >>> index performance and in general how KahaDB compares to something
> >>> like BerkeleyDB.
> >>>
> >>> Thanks
> >>>
> >>>
> >>>
> >>
> >>
> >> There's is some confusion over naming of our persistence options that
> >> doesn't help. There is Kaha - which uses multiple log files and a Hash
> >> based index - this is currently used by the FileCursor - whilst KahaDB
> >> is a newer implementation, which is more robust and typically uses a
> >> BTreeIndex. There is currently a new implementation of the Filecursor
> >> btw - but that's a different matter. You can't currently configure the
> >> HashIndex via the FileCursor -  but it looks like this is the problem
> >> you are encountering - as it looks like you need to increase the max
> >> hash buckets.
> >>
> >>
> >> So I would recommend the following
> >> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
> >> for non-persistent messages - and uses the underlying database for
> >> persistent messages
> >> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> >> problems you are seeing with the Filecursor
> >>
> >> or - increase the maximum number of hash buckets for the FileCursor
> >> index - by setting a Java system property -  maximumCapacity to 65536
> >> (the default is 16384)
> >>
> >> cheers,
> >>
> >> Rob
> >>
> >> http://twitter.com/rajdavies
> >> I work here: http://fusesource.com
> >> My Blog: http://rajdavies.blogspot.com/
> >> I'm writing this: http://www.manning.com/snyder/
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> > --
> > View this message in context:
> >
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message