activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fred Moore <fred.moor...@gmail.com>
Subject Re: OOM with high KahaDB index time
Date Mon, 25 Jan 2010 15:22:10 GMT
Thanks Joe... no I'm still on 5.3 (cannot go 5.4 SNAPSHOT right now), and
wanted to get the best out of the currently available version.

I'll give it a try.
Cheers,
F.


On Mon, Jan 25, 2010 at 4:14 PM, Joe Fernandez <
joe.fernandez@ttmsolutions.com> wrote:

>
> Here it is - its the xml that Dan posted with some slight mod's.  Are you
> using the latest trunk and not the Jan 20 5.4 SNAPSHOT?
>
> <beans
>  xmlns="http://www.springframework.org/schema/beans"
>  xmlns:amq="http://activemq.apache.org/schema/core"
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>  xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>  http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>    <bean
>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>        <property name="locations">
>            <value>file:${activemq.base}/conf/credentials.properties</value>
>        </property>
>    </bean>
>
>    <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="danbroker" dataDirectory="${activemq.base}/data">
>
>        <managementContext>
>            <managementContext createConnector="true"/>
>        </managementContext>
>
>        <persistenceAdapter>
>            <kahaDB directory="${activemq.base}/data/kahadb"/>
>        </persistenceAdapter>
>
>        <destinationPolicy>
>      <policyMap>
>        <policyEntries>
>           <policyEntry queue=">" producerFlowControl="true"
> memoryLimit="10mb">
>           </policyEntry>
>        </policyEntries>
>      </policyMap>
>        </destinationPolicy>
>
>
>        <systemUsage>
>            <systemUsage sendFailIfNoSpace="true">
>            </systemUsage>
>        </systemUsage>
>
>
>        <transportConnectors>
>            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
>        </transportConnectors>
>    </broker>
>    <import resource="jetty.xml"/>
> </beans>
>
>
>
>
> Fred Moore-3 wrote:
> >
> > Hi Joe,
> >
> > can you post your whole activemq.xml for this "good enough" test on 5.3?
> >
> > ...I'm still missing something and getting OOMs.
> > TIA,
> > F.
> >
> >
> > On Fri, Jan 22, 2010 at 9:07 PM, Joe Fernandez <
> > joe.fernandez@ttmsolutions.com> wrote:
> >
> >>
> >> I ran my 5.3 test with the following
> >>
> >>    <systemUsage>
> >>            <systemUsage sendFailIfNoSpace="true">
> >>                <memoryUsage>
> >>                    <memoryUsage limit="100mb"/>
> >>                </memoryUsage>
> >>                <storeUsage>
> >>                    <storeUsage limit="500 mb" name="foo"/>
> >>                 </storeUsage>
> >>                <tempUsage>
> >>                    <tempUsage limit="100mb"/>
> >>                </tempUsage>
> >>            </systemUsage>
> >>        </systemUsage>
> >>
> >> ...and had my producer fill up the store; it tried to pump 400k messages
> >> each at 2k in size. Producer flow control was also enabled. The broker
> >> did
> >> not hurl any OOM exceptions, "just
> javax.jms.ResourceAllocationException:
> >> Usage Manager Store is Full. Stopping producer..." exceptions as
> >> expected.
> >> The producer got kicked off its connection.
> >>
> >> The only time I got OOMs was when I used this.
> >>
> >> <systemUsage>
> >>     <systemUsage sendFailIfNoSpace="true">
> >>     </systemUsage>
> >> </systemUsage>
> >>
> >> Joe
> >>
> >>
> >>
> >> Daniel Kluesing-2 wrote:
> >> >
> >> > So I'm not sure if that's what Rob is talking about being fixed in 5.4
> >> > (And I'll try the snapshot as soon as it's ready) but if I don't have
> >> the
> >> > sendFailIfNoSpace then my understanding is the producers send calls
> >> > block/wait/timeout - as opposed to fail - so it's more difficult to
> get
> >> > into a HA configuration. It's a minor point, not having an OOM is much
> >> > more important, but I definitely want the send calls to fail for the
> >> > producer if the broker ever does anything funny.
> >> >
> >> > Thanks for the feedback on the config, very helpful.
> >> >
> >> > -----Original Message-----
> >> > From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
> >> > Sent: Thursday, January 21, 2010 1:01 PM
> >> > To: users@activemq.apache.org
> >> > Subject: RE: OOM with high KahaDB index time
> >> >
> >> >
> >> > Just for grins, I threw your cfg file into our 5.3 testbed and sure
> >> > enough,
> >> > we got the OOMs; I pumped 200k messages with each being 2k in size.
> >> FWIW,
> >> > taking this out of the cfg file made things run a lot better.
> >> >
> >> >  <systemUsage>
> >> >      <systemUsage sendFailIfNoSpace="true">
> >> >      </systemUsage>
> >> > </systemUsage>
> >> >
> >> > With the above taken out of the cfg file, I was able to pump 400k
> >> messages
> >> > into the broker, no OOMs and memory utilization looked much better. I
> >> also
> >> > gave a fully-defined systemUsage a try and that also appeared to do
> the
> >> > trick.
> >> >
> >> >  <systemUsage>
> >> >             <systemUsage>
> >> >                 <memoryUsage>
> >> >                     <memoryUsage limit="100 mb"/>
> >> >                 </memoryUsage>
> >> >                 <storeUsage>
> >> >                     <storeUsage limit="1 gb" name="foo"/>
> >> >                 </storeUsage>
> >> >                 <tempUsage>
> >> >                     <tempUsage limit="100 mb"/>
> >> >                 </tempUsage>
> >> >             </systemUsage>
> >> >  </systemUsage>
> >> >
> >> > So may be worth giving it a whirl if you can't scoot over to the trunk
> >> and
> >> > ride Rob's patch.
> >> >
> >> > Joe
> >> >
> >> >
> >> > Daniel Kluesing-2 wrote:
> >> >>
> >> >> I tried the suggestion of going with the default cursor, but I still
> >> get
> >> >> OOM errors. I've included my full config file below, I think I'm
> >> running
> >> >> fairly vanilla/default.
> >> >>
> >> >> After about 350k persistent messages, the logs start to look like:
> >> >>
> >> >> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update
> >> took
> >> >> 3118 ms
> >> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> >> took
> >> >> 5118 ms
> >> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> >> took
> >> >> 2736 ms
> >> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> >> took
> >> >> 2945 ms
> >> >>  INFO | Slow KahaDB access: Journal append took: 33 ms, Index update
> >> took
> >> >> 2654 ms
> >> >>  INFO | Slow KahaDB access: Journal append took: 82 ms, Index update
> >> took
> >> >> 3174 ms
> >> >>  INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
> >> took
> >> >> 5891 ms
> >> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> >> took
> >> >> 2906 ms
> >> >>  INFO | Slow KahaDB access: Journal append took: 60 ms, Index update
> >> took
> >> >> 7619 ms
> >> >> Exception in thread "InactivityMonitor WriteCheck"
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >>         at java.util.jar.Attributes.read(Attributes.java:377)
> >> >>         at java.util.jar.Manifest.read(Manifest.java:182)
> >> >>         at java.util.jar.Manifest.<init>(Manifest.java:52)
> >> >>         at
> >> >> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
> >> >>         at java.util.jar.JarFile.getManifest(JarFile.java:146)
> >> >>         at
> >> >> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
> >> >>         at
> >> java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
> >> >>         at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> >> >>         at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> >> >>         at java.security.AccessController.doPrivileged(Native Method)
> >> >>         at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >> >>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >> >>         at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> >> >>         at
> >> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> >> >>         at
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
> >> >>         at
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
> >> >>         at
> >> >>
> >>
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> >> >>         at java.util.TimerThread.mainLoop(Timer.java:512)
> >> >>         at java.util.TimerThread.run(Timer.java:462)
> >> >>
> >> >> Config file:
> >> >>
> >> >> <beans
> >> >>   xmlns="http://www.springframework.org/schema/beans"
> >> >>   xmlns:amq="http://activemq.apache.org/schema/core"
> >> >>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >> >>   xsi:schemaLocation="http://www.springframework.org/schema/beans
> >> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> >> >>   http://activemq.apache.org/schema/core
> >> >> http://activemq.apache.org/schema/core/activemq-core.xsd">
> >> >>
> >> >>     <bean
> >> >>
> >>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> >> >>         <property name="locations">
> >> >>
> >> >> <value>file:${activemq.base}/conf/credentials.properties</value>
> >> >>         </property>
> >> >>     </bean>
> >> >>
> >> >>     <broker xmlns="http://activemq.apache.org/schema/core"
> >> >> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
> >> >>
> >> >>         <managementContext>
> >> >>             <managementContext createConnector="true"/>
> >> >>         </managementContext>
> >> >>
> >> >>         <persistenceAdapter>
> >> >>             <kahaDB directory="${activemq.base}/data/kahadb"/>
> >> >>         </persistenceAdapter>
> >> >>
> >> >>      <destinationPolicy>
> >> >>                      <policyMap>
> >> >>                              <policyEntries>
> >> >>                              <policyEntry queue="P>"
> >> producerFlowControl="true"
> >> >> memoryLimit="10mb"></policyEntry>
> >> >>                      </policyEntries>
> >> >>                      </policyMap>
> >> >>      </destinationPolicy>
> >> >>         <systemUsage>
> >> >>             <systemUsage sendFailIfNoSpace="true">
> >> >>             </systemUsage>
> >> >>         </systemUsage>
> >> >>         <transportConnectors>
> >> >>             <transportConnector name="openwire"
> >> >> uri="tcp://0.0.0.0:61616"/>
> >> >>         </transportConnectors>
> >> >>     </broker>
> >> >>     <import resource="jetty.xml"/>
> >> >> </beans>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Rob Davies [mailto:rajdavies@gmail.com]
> >> >> Sent: Monday, January 18, 2010 10:42 PM
> >> >> To: users@activemq.apache.org
> >> >> Subject: Re: OOM with high KahaDB index time
> >> >>
> >> >>
> >> >> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> I'm running the 5.3 release as a standalone broker. In one case,
a
> >> >>> producer is running without a consumer, producing small, persistent
> >> >>> messages, with the FileCursor pendingQueuePolicy (per
> >> >>> https://issues.apache.org/activemq/browse/AMQ-2512)
> >> >>>  option and flow control memoryLimit set to 100mb for the queue
in
> >> >>> question. (Through a policy entry)
> >> >>>
> >> >>> As the queue grows above 300k messages, KahaDB indexing starts
> >> >>> climbing above 1 second. At around 350k messages, the indexing
is
> >> >>> taking over 8 seconds. At this point, I start getting java out
of
> >> >>> heap space errors in essentially random parts of the code. After
a
> >> >>> while, the producers timeout with a channel inactive for too long
> >> >>> error, and the entire broker basically wedges itself. At this point,
> >> >>> consumers are generally unable to bind to the broker quitting with
> >> >>> timeout errors. When they can connect, consuming a single message
> >> >>> triggers an index re-build, which takes 2-8seconds. Turning on
> >> >>> verbose garbage collection, the jvm is collecting like mad but
> >> >>> reclaiming no space.
> >> >>>
> >> >>> If I restart the broker, it comes back up, I can consume the old
> >> >>> messages, and can handle another 350k messages until it wedges.
> >> >>>
> >> >>> I can reproduce under both default gc and incremental gc.
> >> >>>
> >> >>> Two questions:
> >> >>> - It seems like someone is holding onto a handle to the messages
> >> >>> after they have been persisted to disk - is this a known issue?
> >> >>> Should I open a JIRA for it? (Or is there another explanation?)
> >> >>>
> >> >>> - Is there any documentation about the internals of KahaDB - the
> >> >>> kind of indices etc? I'd like to get a better understanding of
the
> >> >>> index performance and in general how KahaDB compares to something
> >> >>> like BerkeleyDB.
> >> >>>
> >> >>> Thanks
> >> >>>
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >> There's is some confusion over naming of our persistence options that
> >> >> doesn't help. There is Kaha - which uses multiple log files and a
> Hash
> >> >> based index - this is currently used by the FileCursor - whilst
> KahaDB
> >> >> is a newer implementation, which is more robust and typically uses
a
> >> >> BTreeIndex. There is currently a new implementation of the Filecursor
> >> >> btw - but that's a different matter. You can't currently configure
> the
> >> >> HashIndex via the FileCursor -  but it looks like this is the problem
> >> >> you are encountering - as it looks like you need to increase the max
> >> >> hash buckets.
> >> >>
> >> >>
> >> >> So I would recommend the following
> >> >> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
> >> >> for non-persistent messages - and uses the underlying database for
> >> >> persistent messages
> >> >> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> >> >> problems you are seeing with the Filecursor
> >> >>
> >> >> or - increase the maximum number of hash buckets for the FileCursor
> >> >> index - by setting a Java system property -  maximumCapacity to 65536
> >> >> (the default is 16384)
> >> >>
> >> >> cheers,
> >> >>
> >> >> Rob
> >> >>
> >> >> http://twitter.com/rajdavies
> >> >> I work here: http://fusesource.com
> >> >> My Blog: http://rajdavies.blogspot.com/
> >> >> I'm writing this: http://www.manning.com/snyder/
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27307957.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message