activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Fernandez <joe.fernan...@ttmsolutions.com>
Subject Re: OOM with high KahaDB index time
Date Mon, 25 Jan 2010 14:41:09 GMT

Rob,

The trunk (902807)  passed my test; no hurling of OOMs. Memory utilization
under JConsole looked much better. 

My producer didn't get kicked off when the store filled, but I think that's
because in this case it was issuing async sends. 

Joe


rajdavies wrote:
> 
> Hi Joe,
> 
> any chance you can build from trunk and try it ?
> 
> cheers,
> 
> Rob
> 
> On 22 Jan 2010, at 20:07, Joe Fernandez wrote:
> 
>>
>> I ran my 5.3 test with the following
>>
>>    <systemUsage>
>>            <systemUsage sendFailIfNoSpace="true">
>>                <memoryUsage>
>>                    <memoryUsage limit="100mb"/>
>>                </memoryUsage>
>>                <storeUsage>
>>                    <storeUsage limit="500 mb" name="foo"/>
>>                </storeUsage>
>>                <tempUsage>
>>                    <tempUsage limit="100mb"/>
>>                </tempUsage>
>>            </systemUsage>
>>        </systemUsage>
>>
>> ...and had my producer fill up the store; it tried to pump 400k  
>> messages
>> each at 2k in size. Producer flow control was also enabled. The  
>> broker did
>> not hurl any OOM exceptions, "just  
>> javax.jms.ResourceAllocationException:
>> Usage Manager Store is Full. Stopping producer..." exceptions as  
>> expected.
>> The producer got kicked off its connection.
>>
>> The only time I got OOMs was when I used this.
>>
>> <systemUsage>
>>     <systemUsage sendFailIfNoSpace="true">
>>     </systemUsage>
>> </systemUsage>
>>
>> Joe
>>
>>
>>
>> Daniel Kluesing-2 wrote:
>>>
>>> So I'm not sure if that's what Rob is talking about being fixed in  
>>> 5.4
>>> (And I'll try the snapshot as soon as it's ready) but if I don't  
>>> have the
>>> sendFailIfNoSpace then my understanding is the producers send calls
>>> block/wait/timeout - as opposed to fail - so it's more difficult to  
>>> get
>>> into a HA configuration. It's a minor point, not having an OOM is  
>>> much
>>> more important, but I definitely want the send calls to fail for the
>>> producer if the broker ever does anything funny.
>>>
>>> Thanks for the feedback on the config, very helpful.
>>>
>>> -----Original Message-----
>>> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>>> Sent: Thursday, January 21, 2010 1:01 PM
>>> To: users@activemq.apache.org
>>> Subject: RE: OOM with high KahaDB index time
>>>
>>>
>>> Just for grins, I threw your cfg file into our 5.3 testbed and sure
>>> enough,
>>> we got the OOMs; I pumped 200k messages with each being 2k in size.  
>>> FWIW,
>>> taking this out of the cfg file made things run a lot better.
>>>
>>> <systemUsage>
>>>     <systemUsage sendFailIfNoSpace="true">
>>>     </systemUsage>
>>> </systemUsage>
>>>
>>> With the above taken out of the cfg file, I was able to pump 400k  
>>> messages
>>> into the broker, no OOMs and memory utilization looked much better.  
>>> I also
>>> gave a fully-defined systemUsage a try and that also appeared to do  
>>> the
>>> trick.
>>>
>>> <systemUsage>
>>>            <systemUsage>
>>>                <memoryUsage>
>>>                    <memoryUsage limit="100 mb"/>
>>>                </memoryUsage>
>>>                <storeUsage>
>>>                    <storeUsage limit="1 gb" name="foo"/>
>>>                </storeUsage>
>>>                <tempUsage>
>>>                    <tempUsage limit="100 mb"/>
>>>                </tempUsage>
>>>            </systemUsage>
>>> </systemUsage>
>>>
>>> So may be worth giving it a whirl if you can't scoot over to the  
>>> trunk and
>>> ride Rob's patch.
>>>
>>> Joe
>>>
>>>
>>> Daniel Kluesing-2 wrote:
>>>>
>>>> I tried the suggestion of going with the default cursor, but I  
>>>> still get
>>>> OOM errors. I've included my full config file below, I think I'm  
>>>> running
>>>> fairly vanilla/default.
>>>>
>>>> After about 350k persistent messages, the logs start to look like:
>>>>
>>>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index  
>>>> update took
>>>> 3118 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
>>>> took
>>>> 5118 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
>>>> took
>>>> 2736 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
>>>> took
>>>> 2945 ms
>>>> INFO | Slow KahaDB access: Journal append took: 33 ms, Index  
>>>> update took
>>>> 2654 ms
>>>> INFO | Slow KahaDB access: Journal append took: 82 ms, Index  
>>>> update took
>>>> 3174 ms
>>>> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update  
>>>> took
>>>> 5891 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update  
>>>> took
>>>> 2906 ms
>>>> INFO | Slow KahaDB access: Journal append took: 60 ms, Index  
>>>> update took
>>>> 7619 ms
>>>> Exception in thread "InactivityMonitor WriteCheck"
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>        at java.util.jar.Attributes.read(Attributes.java:377)
>>>>        at java.util.jar.Manifest.read(Manifest.java:182)
>>>>        at java.util.jar.Manifest.<init>(Manifest.java:52)
>>>>        at
>>>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>>>>        at java.util.jar.JarFile.getManifest(JarFile.java:146)
>>>>        at
>>>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>>>>        at java.net.URLClassLoader.defineClass(URLClassLoader.java: 
>>>> 221)
>>>>        at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>>>>        at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
>>>> 319)
>>>>        at
>>>> org 
>>>> .apache 
>>>> .activemq 
>>>> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>>>>        at
>>>> org.apache.activemq.transport.InactivityMonitor 
>>>> $2.run(InactivityMonitor.java:106)
>>>>        at
>>>> org 
>>>> .apache 
>>>> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>>>>        at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>        at java.util.TimerThread.run(Timer.java:462)
>>>>
>>>> Config file:
>>>>
>>>> <beans
>>>>  xmlns="http://www.springframework.org/schema/beans"
>>>>  xmlns:amq="http://activemq.apache.org/schema/core"
>>>>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>  xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>>  http://activemq.apache.org/schema/core
>>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>>
>>>>    <bean
>>>> class 
>>>> = 
>>>> "org 
>>>> .springframework 
>>>> .beans.factory.config.PropertyPlaceholderConfigurer">
>>>>        <property name="locations">
>>>>
>>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>>>        </property>
>>>>    </bean>
>>>>
>>>>    <broker xmlns="http://activemq.apache.org/schema/core"
>>>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>>>
>>>>        <managementContext>
>>>>            <managementContext createConnector="true"/>
>>>>        </managementContext>
>>>>
>>>>        <persistenceAdapter>
>>>>            <kahaDB directory="${activemq.base}/data/kahadb"/>
>>>>        </persistenceAdapter>
>>>>
>>>> 	<destinationPolicy>
>>>>      		<policyMap>
>>>> 				<policyEntries>
>>>>          			<policyEntry queue="P>" producerFlowControl="true"
>>>> memoryLimit="10mb"></policyEntry>
>>>>        		</policyEntries>
>>>>      		</policyMap>
>>>> 	</destinationPolicy>
>>>>        <systemUsage>
>>>>            <systemUsage sendFailIfNoSpace="true">
>>>>            </systemUsage>
>>>>        </systemUsage>
>>>>        <transportConnectors>
>>>>            <transportConnector name="openwire"
>>>> uri="tcp://0.0.0.0:61616"/>
>>>>        </transportConnectors>
>>>>    </broker>
>>>>    <import resource="jetty.xml"/>
>>>> </beans>
>>>>
>>>> -----Original Message-----
>>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>>> Sent: Monday, January 18, 2010 10:42 PM
>>>> To: users@activemq.apache.org
>>>> Subject: Re: OOM with high KahaDB index time
>>>>
>>>>
>>>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm running the 5.3 release as a standalone broker. In one case, a
>>>>> producer is running without a consumer, producing small, persistent
>>>>> messages, with the FileCursor pendingQueuePolicy (per
>>>>> https://issues.apache.org/activemq/browse/AMQ-2512)
>>>>> option and flow control memoryLimit set to 100mb for the queue in
>>>>> question. (Through a policy entry)
>>>>>
>>>>> As the queue grows above 300k messages, KahaDB indexing starts
>>>>> climbing above 1 second. At around 350k messages, the indexing is
>>>>> taking over 8 seconds. At this point, I start getting java out of
>>>>> heap space errors in essentially random parts of the code. After a
>>>>> while, the producers timeout with a channel inactive for too long
>>>>> error, and the entire broker basically wedges itself. At this  
>>>>> point,
>>>>> consumers are generally unable to bind to the broker quitting with
>>>>> timeout errors. When they can connect, consuming a single message
>>>>> triggers an index re-build, which takes 2-8seconds. Turning on
>>>>> verbose garbage collection, the jvm is collecting like mad but
>>>>> reclaiming no space.
>>>>>
>>>>> If I restart the broker, it comes back up, I can consume the old
>>>>> messages, and can handle another 350k messages until it wedges.
>>>>>
>>>>> I can reproduce under both default gc and incremental gc.
>>>>>
>>>>> Two questions:
>>>>> - It seems like someone is holding onto a handle to the messages
>>>>> after they have been persisted to disk - is this a known issue?
>>>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>>>
>>>>> - Is there any documentation about the internals of KahaDB - the
>>>>> kind of indices etc? I'd like to get a better understanding of the
>>>>> index performance and in general how KahaDB compares to something
>>>>> like BerkeleyDB.
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> There's is some confusion over naming of our persistence options  
>>>> that
>>>> doesn't help. There is Kaha - which uses multiple log files and a  
>>>> Hash
>>>> based index - this is currently used by the FileCursor - whilst  
>>>> KahaDB
>>>> is a newer implementation, which is more robust and typically uses a
>>>> BTreeIndex. There is currently a new implementation of the  
>>>> Filecursor
>>>> btw - but that's a different matter. You can't currently configure  
>>>> the
>>>> HashIndex via the FileCursor -  but it looks like this is the  
>>>> problem
>>>> you are encountering - as it looks like you need to increase the max
>>>> hash buckets.
>>>>
>>>>
>>>> So I would recommend the following
>>>> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
>>>> for non-persistent messages - and uses the underlying database for
>>>> persistent messages
>>>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>>>> problems you are seeing with the Filecursor
>>>>
>>>> or - increase the maximum number of hash buckets for the FileCursor
>>>> index - by setting a Java system property -  maximumCapacity to  
>>>> 65536
>>>> (the default is 16384)
>>>>
>>>> cheers,
>>>>
>>>> Rob
>>>>
>>>> http://twitter.com/rajdavies
>>>> I work here: http://fusesource.com
>>>> My Blog: http://rajdavies.blogspot.com/
>>>> I'm writing this: http://www.manning.com/snyder/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> -- 
>>> View this message in context:
>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
> 
> 
> 
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27307657.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message