activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Fernandez <joe.fernan...@ttmsolutions.com>
Subject RE: OOM with high KahaDB index time
Date Fri, 22 Jan 2010 20:07:03 GMT

I ran my 5.3 test with the following 

    <systemUsage>
            <systemUsage sendFailIfNoSpace="true">            
                <memoryUsage>
                    <memoryUsage limit="100mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="500 mb" name="foo"/> 
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="100mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

...and had my producer fill up the store; it tried to pump 400k messages
each at 2k in size. Producer flow control was also enabled. The broker did
not hurl any OOM exceptions, "just javax.jms.ResourceAllocationException:
Usage Manager Store is Full. Stopping producer..." exceptions as expected.
The producer got kicked off its connection.

The only time I got OOMs was when I used this.

<systemUsage>
     <systemUsage sendFailIfNoSpace="true">
     </systemUsage>
</systemUsage>

Joe

 

Daniel Kluesing-2 wrote:
> 
> So I'm not sure if that's what Rob is talking about being fixed in 5.4
> (And I'll try the snapshot as soon as it's ready) but if I don't have the
> sendFailIfNoSpace then my understanding is the producers send calls
> block/wait/timeout - as opposed to fail - so it's more difficult to get
> into a HA configuration. It's a minor point, not having an OOM is much
> more important, but I definitely want the send calls to fail for the
> producer if the broker ever does anything funny.
> 
> Thanks for the feedback on the config, very helpful.
> 
> -----Original Message-----
> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com] 
> Sent: Thursday, January 21, 2010 1:01 PM
> To: users@activemq.apache.org
> Subject: RE: OOM with high KahaDB index time
> 
> 
> Just for grins, I threw your cfg file into our 5.3 testbed and sure
> enough,
> we got the OOMs; I pumped 200k messages with each being 2k in size. FWIW,
> taking this out of the cfg file made things run a lot better. 
> 
>  <systemUsage>
>      <systemUsage sendFailIfNoSpace="true">
>      </systemUsage>
> </systemUsage>
> 
> With the above taken out of the cfg file, I was able to pump 400k messages
> into the broker, no OOMs and memory utilization looked much better. I also
> gave a fully-defined systemUsage a try and that also appeared to do the
> trick. 
> 
>  <systemUsage>
>             <systemUsage>
>                 <memoryUsage>
>                     <memoryUsage limit="100 mb"/>
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="1 gb" name="foo"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="100 mb"/>
>                 </tempUsage>
>             </systemUsage>
>  </systemUsage>
> 
> So may be worth giving it a whirl if you can't scoot over to the trunk and
> ride Rob's patch.  
> 
> Joe   
>  
> 
> Daniel Kluesing-2 wrote:
>> 
>> I tried the suggestion of going with the default cursor, but I still get
>> OOM errors. I've included my full config file below, I think I'm running
>> fairly vanilla/default.
>> 
>> After about 350k persistent messages, the logs start to look like:
>> 
>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took
>> 3118 ms
>>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
>> 5118 ms
>>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
>> 2736 ms
>>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
>> 2945 ms
>>  INFO | Slow KahaDB access: Journal append took: 33 ms, Index update took
>> 2654 ms
>>  INFO | Slow KahaDB access: Journal append took: 82 ms, Index update took
>> 3174 ms
>>  INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took
>> 5891 ms
>>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
>> 2906 ms
>>  INFO | Slow KahaDB access: Journal append took: 60 ms, Index update took
>> 7619 ms
>> Exception in thread "InactivityMonitor WriteCheck"
>> java.lang.OutOfMemoryError: Java heap space
>>         at java.util.jar.Attributes.read(Attributes.java:377)
>>         at java.util.jar.Manifest.read(Manifest.java:182)
>>         at java.util.jar.Manifest.<init>(Manifest.java:52)
>>         at
>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>>         at java.util.jar.JarFile.getManifest(JarFile.java:146)
>>         at
>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
>>         at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>         at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>>         at
>> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>>         at
>> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
>>         at
>> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>>         at java.util.TimerThread.mainLoop(Timer.java:512)
>>         at java.util.TimerThread.run(Timer.java:462)
>> 
>> Config file:
>> 
>> <beans
>>   xmlns="http://www.springframework.org/schema/beans"
>>   xmlns:amq="http://activemq.apache.org/schema/core"
>>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>   xsi:schemaLocation="http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>   http://activemq.apache.org/schema/core
>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>> 
>>     <bean
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>>         <property name="locations">
>>            
>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>         </property>      
>>     </bean>
>> 
>>     <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>         
>>         <managementContext>
>>             <managementContext createConnector="true"/>
>>         </managementContext>
>> 
>>         <persistenceAdapter>
>>             <kahaDB directory="${activemq.base}/data/kahadb"/>
>>         </persistenceAdapter>
>>    
>> 	<destinationPolicy>
>>       		<policyMap>
>> 				<policyEntries>
>>           			<policyEntry queue="P>" producerFlowControl="true"
>> memoryLimit="10mb"></policyEntry>
>>         		</policyEntries>
>>       		</policyMap>
>> 	</destinationPolicy>     
>>         <systemUsage>
>>             <systemUsage sendFailIfNoSpace="true">
>>             </systemUsage>
>>         </systemUsage>
>>         <transportConnectors>
>>             <transportConnector name="openwire"
>> uri="tcp://0.0.0.0:61616"/>
>>         </transportConnectors>
>>     </broker>
>>     <import resource="jetty.xml"/>
>> </beans>
>> 
>> -----Original Message-----
>> From: Rob Davies [mailto:rajdavies@gmail.com] 
>> Sent: Monday, January 18, 2010 10:42 PM
>> To: users@activemq.apache.org
>> Subject: Re: OOM with high KahaDB index time
>> 
>> 
>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>> 
>>> Hi,
>>>
>>> I'm running the 5.3 release as a standalone broker. In one case, a  
>>> producer is running without a consumer, producing small, persistent  
>>> messages, with the FileCursor pendingQueuePolicy (per
>>> https://issues.apache.org/activemq/browse/AMQ-2512) 
>>>  option and flow control memoryLimit set to 100mb for the queue in  
>>> question. (Through a policy entry)
>>>
>>> As the queue grows above 300k messages, KahaDB indexing starts  
>>> climbing above 1 second. At around 350k messages, the indexing is  
>>> taking over 8 seconds. At this point, I start getting java out of  
>>> heap space errors in essentially random parts of the code. After a  
>>> while, the producers timeout with a channel inactive for too long  
>>> error, and the entire broker basically wedges itself. At this point,  
>>> consumers are generally unable to bind to the broker quitting with  
>>> timeout errors. When they can connect, consuming a single message  
>>> triggers an index re-build, which takes 2-8seconds. Turning on  
>>> verbose garbage collection, the jvm is collecting like mad but  
>>> reclaiming no space.
>>>
>>> If I restart the broker, it comes back up, I can consume the old  
>>> messages, and can handle another 350k messages until it wedges.
>>>
>>> I can reproduce under both default gc and incremental gc.
>>>
>>> Two questions:
>>> - It seems like someone is holding onto a handle to the messages  
>>> after they have been persisted to disk - is this a known issue?  
>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>
>>> - Is there any documentation about the internals of KahaDB - the  
>>> kind of indices etc? I'd like to get a better understanding of the  
>>> index performance and in general how KahaDB compares to something  
>>> like BerkeleyDB.
>>>
>>> Thanks
>>>
>>>
>>>
>> 
>> 
>> There's is some confusion over naming of our persistence options that  
>> doesn't help. There is Kaha - which uses multiple log files and a Hash  
>> based index - this is currently used by the FileCursor - whilst KahaDB  
>> is a newer implementation, which is more robust and typically uses a  
>> BTreeIndex. There is currently a new implementation of the Filecursor  
>> btw - but that's a different matter. You can't currently configure the  
>> HashIndex via the FileCursor -  but it looks like this is the problem  
>> you are encountering - as it looks like you need to increase the max  
>> hash buckets.
>> 
>> 
>> So I would recommend the following
>> 1. Use the default pendingQueuePolicy (which only uses a FileCursor  
>> for non-persistent messages - and uses the underlying database for  
>> persistent messages
>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the  
>> problems you are seeing with the Filecursor
>> 
>> or - increase the maximum number of hash buckets for the FileCursor  
>> index - by setting a Java system property -  maximumCapacity to 65536  
>> (the default is 16384)
>> 
>> cheers,
>> 
>> Rob
>> 
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>> 
>> 
>> 
>> 
>> 
>> 
>> 
> 
> -- 
> View this message in context:
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message