activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: OOM with high KahaDB index time
Date Fri, 22 Jan 2010 20:26:49 GMT
Hi Joe,

any chance you can build from trunk and try it ?

cheers,

Rob

On 22 Jan 2010, at 20:07, Joe Fernandez wrote:

>
> I ran my 5.3 test with the following
>
>    <systemUsage>
>            <systemUsage sendFailIfNoSpace="true">
>                <memoryUsage>
>                    <memoryUsage limit="100mb"/>
>                </memoryUsage>
>                <storeUsage>
>                    <storeUsage limit="500 mb" name="foo"/>
>                </storeUsage>
>                <tempUsage>
>                    <tempUsage limit="100mb"/>
>                </tempUsage>
>            </systemUsage>
>        </systemUsage>
>
> ...and had my producer fill up the store; it tried to pump 400k  
> messages
> each at 2k in size. Producer flow control was also enabled. The  
> broker did
> not hurl any OOM exceptions, "just  
> javax.jms.ResourceAllocationException:
> Usage Manager Store is Full. Stopping producer..." exceptions as  
> expected.
> The producer got kicked off its connection.
>
> The only time I got OOMs was when I used this.
>
> <systemUsage>
>     <systemUsage sendFailIfNoSpace="true">
>     </systemUsage>
> </systemUsage>
>
> Joe
>
>
>
> Daniel Kluesing-2 wrote:
>>
>> So I'm not sure if that's what Rob is talking about being fixed in  
>> 5.4
>> (And I'll try the snapshot as soon as it's ready) but if I don't  
>> have the
>> sendFailIfNoSpace then my understanding is the producers send calls
>> block/wait/timeout - as opposed to fail - so it's more difficult to  
>> get
>> into a HA configuration. It's a minor point, not having an OOM is  
>> much
>> more important, but I definitely want the send calls to fail for the
>> producer if the broker ever does anything funny.
>>
>> Thanks for the feedback on the config, very helpful.
>>
>> -----Original Message-----
>> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>> Sent: Thursday, January 21, 2010 1:01 PM
>> To: users@activemq.apache.org
>> Subject: RE: OOM with high KahaDB index time
>>
>>
>> Just for grins, I threw your cfg file into our 5.3 testbed and sure
>> enough,
>> we got the OOMs; I pumped 200k messages with each being 2k in size.  
>> FWIW,
>> taking this out of the cfg file made things run a lot better.
>>
>> <systemUsage>
>>     <systemUsage sendFailIfNoSpace="true">
>>     </systemUsage>
>> </systemUsage>
>>
>> With the above taken out of the cfg file, I was able to pump 400k  
>> messages
>> into the broker, no OOMs and memory utilization looked much better.  
>> I also
>> gave a fully-defined systemUsage a try and that also appeared to do  
>> the
>> trick.
>>
>> <systemUsage>
>>            <systemUsage>
>>                <memoryUsage>
>>                    <memoryUsage limit="100 mb"/>
>>                </memoryUsage>
>>                <storeUsage>
>>                    <storeUsage limit="1 gb" name="foo"/>
>>                </storeUsage>
>>                <tempUsage>
>>                    <tempUsage limit="100 mb"/>
>>                </tempUsage>
>>            </systemUsage>
>> </systemUsage>
>>
>> So may be worth giving it a whirl if you can't scoot over to the  
>> trunk and
>> ride Rob's patch.
>>
>> Joe
>>
>>
>> Daniel Kluesing-2 wrote:
>>>
>>> I tried the suggestion of going with the default cursor, but I  
>>> still get
>>> OOM errors. I've included my full config file below, I think I'm  
>>> running
>>> fairly vanilla/default.
>>>
>>> After about 350k persistent messages, the logs start to look like:
>>>
>>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index  
>>> update took
>>> 3118 ms
>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
>>> took
>>> 5118 ms
>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
>>> took
>>> 2736 ms
>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
>>> took
>>> 2945 ms
>>> INFO | Slow KahaDB access: Journal append took: 33 ms, Index  
>>> update took
>>> 2654 ms
>>> INFO | Slow KahaDB access: Journal append took: 82 ms, Index  
>>> update took
>>> 3174 ms
>>> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update  
>>> took
>>> 5891 ms
>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
>>> took
>>> 2906 ms
>>> INFO | Slow KahaDB access: Journal append took: 60 ms, Index  
>>> update took
>>> 7619 ms
>>> Exception in thread "InactivityMonitor WriteCheck"
>>> java.lang.OutOfMemoryError: Java heap space
>>>        at java.util.jar.Attributes.read(Attributes.java:377)
>>>        at java.util.jar.Manifest.read(Manifest.java:182)
>>>        at java.util.jar.Manifest.<init>(Manifest.java:52)
>>>        at
>>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>>>        at java.util.jar.JarFile.getManifest(JarFile.java:146)
>>>        at
>>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>>>        at java.net.URLClassLoader.defineClass(URLClassLoader.java: 
>>> 221)
>>>        at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>>>        at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
>>> 319)
>>>        at
>>> org 
>>> .apache 
>>> .activemq 
>>> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>>>        at
>>> org.apache.activemq.transport.InactivityMonitor 
>>> $2.run(InactivityMonitor.java:106)
>>>        at
>>> org 
>>> .apache 
>>> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>>>        at java.util.TimerThread.mainLoop(Timer.java:512)
>>>        at java.util.TimerThread.run(Timer.java:462)
>>>
>>> Config file:
>>>
>>> <beans
>>>  xmlns="http://www.springframework.org/schema/beans"
>>>  xmlns:amq="http://activemq.apache.org/schema/core"
>>>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>  xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>  http://activemq.apache.org/schema/core
>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>
>>>    <bean
>>> class 
>>> = 
>>> "org 
>>> .springframework 
>>> .beans.factory.config.PropertyPlaceholderConfigurer">
>>>        <property name="locations">
>>>
>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>>        </property>
>>>    </bean>
>>>
>>>    <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>>
>>>        <managementContext>
>>>            <managementContext createConnector="true"/>
>>>        </managementContext>
>>>
>>>        <persistenceAdapter>
>>>            <kahaDB directory="${activemq.base}/data/kahadb"/>
>>>        </persistenceAdapter>
>>>
>>> 	<destinationPolicy>
>>>      		<policyMap>
>>> 				<policyEntries>
>>>          			<policyEntry queue="P>" producerFlowControl="true"
>>> memoryLimit="10mb"></policyEntry>
>>>        		</policyEntries>
>>>      		</policyMap>
>>> 	</destinationPolicy>
>>>        <systemUsage>
>>>            <systemUsage sendFailIfNoSpace="true">
>>>            </systemUsage>
>>>        </systemUsage>
>>>        <transportConnectors>
>>>            <transportConnector name="openwire"
>>> uri="tcp://0.0.0.0:61616"/>
>>>        </transportConnectors>
>>>    </broker>
>>>    <import resource="jetty.xml"/>
>>> </beans>
>>>
>>> -----Original Message-----
>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>> Sent: Monday, January 18, 2010 10:42 PM
>>> To: users@activemq.apache.org
>>> Subject: Re: OOM with high KahaDB index time
>>>
>>>
>>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm running the 5.3 release as a standalone broker. In one case, a
>>>> producer is running without a consumer, producing small, persistent
>>>> messages, with the FileCursor pendingQueuePolicy (per
>>>> https://issues.apache.org/activemq/browse/AMQ-2512)
>>>> option and flow control memoryLimit set to 100mb for the queue in
>>>> question. (Through a policy entry)
>>>>
>>>> As the queue grows above 300k messages, KahaDB indexing starts
>>>> climbing above 1 second. At around 350k messages, the indexing is
>>>> taking over 8 seconds. At this point, I start getting java out of
>>>> heap space errors in essentially random parts of the code. After a
>>>> while, the producers timeout with a channel inactive for too long
>>>> error, and the entire broker basically wedges itself. At this  
>>>> point,
>>>> consumers are generally unable to bind to the broker quitting with
>>>> timeout errors. When they can connect, consuming a single message
>>>> triggers an index re-build, which takes 2-8seconds. Turning on
>>>> verbose garbage collection, the jvm is collecting like mad but
>>>> reclaiming no space.
>>>>
>>>> If I restart the broker, it comes back up, I can consume the old
>>>> messages, and can handle another 350k messages until it wedges.
>>>>
>>>> I can reproduce under both default gc and incremental gc.
>>>>
>>>> Two questions:
>>>> - It seems like someone is holding onto a handle to the messages
>>>> after they have been persisted to disk - is this a known issue?
>>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>>
>>>> - Is there any documentation about the internals of KahaDB - the
>>>> kind of indices etc? I'd like to get a better understanding of the
>>>> index performance and in general how KahaDB compares to something
>>>> like BerkeleyDB.
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>
>>>
>>> There's is some confusion over naming of our persistence options  
>>> that
>>> doesn't help. There is Kaha - which uses multiple log files and a  
>>> Hash
>>> based index - this is currently used by the FileCursor - whilst  
>>> KahaDB
>>> is a newer implementation, which is more robust and typically uses a
>>> BTreeIndex. There is currently a new implementation of the  
>>> Filecursor
>>> btw - but that's a different matter. You can't currently configure  
>>> the
>>> HashIndex via the FileCursor -  but it looks like this is the  
>>> problem
>>> you are encountering - as it looks like you need to increase the max
>>> hash buckets.
>>>
>>>
>>> So I would recommend the following
>>> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
>>> for non-persistent messages - and uses the underlying database for
>>> persistent messages
>>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>>> problems you are seeing with the Filecursor
>>>
>>> or - increase the maximum number of hash buckets for the FileCursor
>>> index - by setting a Java system property -  maximumCapacity to  
>>> 65536
>>> (the default is 16384)
>>>
>>> cheers,
>>>
>>> Rob
>>>
>>> http://twitter.com/rajdavies
>>> I work here: http://fusesource.com
>>> My Blog: http://rajdavies.blogspot.com/
>>> I'm writing this: http://www.manning.com/snyder/
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>
> -- 
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/






Mime
View raw message