activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: OOM with high KahaDB index time
Date Thu, 21 Jan 2010 20:12:34 GMT
Hi Daniel,

there's a potential cut for this in trunk - the test case passes - but  
would like others to try it out before closing the issue.  
Unfortunately latest 5.4-SNAPSHOT hasn't been built yet - but when it  
is - please give it a try

thanks,

Rob
On 21 Jan 2010, at 19:45, Daniel Kluesing wrote:

> I tried the suggestion of going with the default cursor, but I still  
> get OOM errors. I've included my full config file below, I think I'm  
> running fairly vanilla/default.
>
> After about 350k persistent messages, the logs start to look like:
>
> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update  
> took 3118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
> took 5118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
> took 2736 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
> took 2945 ms
> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update  
> took 2654 ms
> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update  
> took 3174 ms
> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update  
> took 5891 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
> took 2906 ms
> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update  
> took 7619 ms
> Exception in thread "InactivityMonitor WriteCheck"  
> java.lang.OutOfMemoryError: Java heap space
>        at java.util.jar.Attributes.read(Attributes.java:377)
>        at java.util.jar.Manifest.read(Manifest.java:182)
>        at java.util.jar.Manifest.<init>(Manifest.java:52)
>        at  
> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>        at java.util.jar.JarFile.getManifest(JarFile.java:146)
>        at sun.misc.URLClassPath$JarLoader 
> $2.getManifest(URLClassPath.java:693)
>        at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
>        at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
> 319)
>        at  
> org 
> .apache 
> .activemq 
> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>        at org.apache.activemq.transport.InactivityMonitor 
> $2.run(InactivityMonitor.java:106)
>        at  
> org 
> .apache 
> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>        at java.util.TimerThread.mainLoop(Timer.java:512)
>        at java.util.TimerThread.run(Timer.java:462)
>
> Config file:
>
> <beans
>  xmlns="http://www.springframework.org/schema/beans"
>  xmlns:amq="http://activemq.apache.org/schema/core"
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd

> ">
>
>    <bean  
> class 
> = 
> "org 
> .springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>        <property name="locations">
>            <value>file:${activemq.base}/conf/credentials.properties</ 
> value>
>        </property>
>    </bean>
>
>    <broker xmlns="http://activemq.apache.org/schema/core"  
> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>
>        <managementContext>
>            <managementContext createConnector="true"/>
>        </managementContext>
>
>        <persistenceAdapter>
>            <kahaDB directory="${activemq.base}/data/kahadb"/>
>        </persistenceAdapter>
>
> 	<destinationPolicy>
>      		<policyMap>
> 				<policyEntries>
>          			<policyEntry queue="P>" producerFlowControl="true"  
> memoryLimit="10mb"></policyEntry>
>        		</policyEntries>
>      		</policyMap>
> 	</destinationPolicy>
>        <systemUsage>
>            <systemUsage sendFailIfNoSpace="true">
>            </systemUsage>
>        </systemUsage>
>        <transportConnectors>
>            <transportConnector name="openwire" uri="tcp:// 
> 0.0.0.0:61616"/>
>        </transportConnectors>
>    </broker>
>    <import resource="jetty.xml"/>
> </beans>
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Monday, January 18, 2010 10:42 PM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
>
> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>
>> Hi,
>>
>> I'm running the 5.3 release as a standalone broker. In one case, a
>> producer is running without a consumer, producing small, persistent
>> messages, with the FileCursor pendingQueuePolicy (per https://issues.apache.org/activemq/browse/AMQ-2512)
>> option and flow control memoryLimit set to 100mb for the queue in
>> question. (Through a policy entry)
>>
>> As the queue grows above 300k messages, KahaDB indexing starts
>> climbing above 1 second. At around 350k messages, the indexing is
>> taking over 8 seconds. At this point, I start getting java out of
>> heap space errors in essentially random parts of the code. After a
>> while, the producers timeout with a channel inactive for too long
>> error, and the entire broker basically wedges itself. At this point,
>> consumers are generally unable to bind to the broker quitting with
>> timeout errors. When they can connect, consuming a single message
>> triggers an index re-build, which takes 2-8seconds. Turning on
>> verbose garbage collection, the jvm is collecting like mad but
>> reclaiming no space.
>>
>> If I restart the broker, it comes back up, I can consume the old
>> messages, and can handle another 350k messages until it wedges.
>>
>> I can reproduce under both default gc and incremental gc.
>>
>> Two questions:
>> - It seems like someone is holding onto a handle to the messages
>> after they have been persisted to disk - is this a known issue?
>> Should I open a JIRA for it? (Or is there another explanation?)
>>
>> - Is there any documentation about the internals of KahaDB - the
>> kind of indices etc? I'd like to get a better understanding of the
>> index performance and in general how KahaDB compares to something
>> like BerkeleyDB.
>>
>> Thanks
>>
>>
>>
>
>
> There's is some confusion over naming of our persistence options that
> doesn't help. There is Kaha - which uses multiple log files and a Hash
> based index - this is currently used by the FileCursor - whilst KahaDB
> is a newer implementation, which is more robust and typically uses a
> BTreeIndex. There is currently a new implementation of the Filecursor
> btw - but that's a different matter. You can't currently configure the
> HashIndex via the FileCursor -  but it looks like this is the problem
> you are encountering - as it looks like you need to increase the max
> hash buckets.
>
>
> So I would recommend the following
> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
> for non-persistent messages - and uses the underlying database for
> persistent messages
> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> problems you are seeing with the Filecursor
>
> or - increase the maximum number of hash buckets for the FileCursor
> index - by setting a Java system property -  maximumCapacity to 65536
> (the default is 16384)
>
> cheers,
>
> Rob
>
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>

Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/






Mime
View raw message