activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: OOM with high KahaDB index time
Date Fri, 22 Jan 2010 23:11:53 GMT
Florida! - nice
On 22 Jan 2010, at 21:57, Joe Fernandez wrote:

>
> Its Miller-time here in Florida, so I'll give it a go on Monday d:)
>
> Joe
>
>
> rajdavies wrote:
>>
>> Hi Joe,
>>
>> any chance you can build from trunk and try it ?
>>
>> cheers,
>>
>> Rob
>>
>> On 22 Jan 2010, at 20:07, Joe Fernandez wrote:
>>
>>>
>>> I ran my 5.3 test with the following
>>>
>>>   <systemUsage>
>>>           <systemUsage sendFailIfNoSpace="true">
>>>               <memoryUsage>
>>>                   <memoryUsage limit="100mb"/>
>>>               </memoryUsage>
>>>               <storeUsage>
>>>                   <storeUsage limit="500 mb" name="foo"/>
>>>               </storeUsage>
>>>               <tempUsage>
>>>                   <tempUsage limit="100mb"/>
>>>               </tempUsage>
>>>           </systemUsage>
>>>       </systemUsage>
>>>
>>> ...and had my producer fill up the store; it tried to pump 400k
>>> messages
>>> each at 2k in size. Producer flow control was also enabled. The
>>> broker did
>>> not hurl any OOM exceptions, "just
>>> javax.jms.ResourceAllocationException:
>>> Usage Manager Store is Full. Stopping producer..." exceptions as
>>> expected.
>>> The producer got kicked off its connection.
>>>
>>> The only time I got OOMs was when I used this.
>>>
>>> <systemUsage>
>>>    <systemUsage sendFailIfNoSpace="true">
>>>    </systemUsage>
>>> </systemUsage>
>>>
>>> Joe
>>>
>>>
>>>
>>> Daniel Kluesing-2 wrote:
>>>>
>>>> So I'm not sure if that's what Rob is talking about being fixed in
>>>> 5.4
>>>> (And I'll try the snapshot as soon as it's ready) but if I don't
>>>> have the
>>>> sendFailIfNoSpace then my understanding is the producers send calls
>>>> block/wait/timeout - as opposed to fail - so it's more difficult to
>>>> get
>>>> into a HA configuration. It's a minor point, not having an OOM is
>>>> much
>>>> more important, but I definitely want the send calls to fail for  
>>>> the
>>>> producer if the broker ever does anything funny.
>>>>
>>>> Thanks for the feedback on the config, very helpful.
>>>>
>>>> -----Original Message-----
>>>> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>>>> Sent: Thursday, January 21, 2010 1:01 PM
>>>> To: users@activemq.apache.org
>>>> Subject: RE: OOM with high KahaDB index time
>>>>
>>>>
>>>> Just for grins, I threw your cfg file into our 5.3 testbed and sure
>>>> enough,
>>>> we got the OOMs; I pumped 200k messages with each being 2k in size.
>>>> FWIW,
>>>> taking this out of the cfg file made things run a lot better.
>>>>
>>>> <systemUsage>
>>>>    <systemUsage sendFailIfNoSpace="true">
>>>>    </systemUsage>
>>>> </systemUsage>
>>>>
>>>> With the above taken out of the cfg file, I was able to pump 400k
>>>> messages
>>>> into the broker, no OOMs and memory utilization looked much better.
>>>> I also
>>>> gave a fully-defined systemUsage a try and that also appeared to do
>>>> the
>>>> trick.
>>>>
>>>> <systemUsage>
>>>>           <systemUsage>
>>>>               <memoryUsage>
>>>>                   <memoryUsage limit="100 mb"/>
>>>>               </memoryUsage>
>>>>               <storeUsage>
>>>>                   <storeUsage limit="1 gb" name="foo"/>
>>>>               </storeUsage>
>>>>               <tempUsage>
>>>>                   <tempUsage limit="100 mb"/>
>>>>               </tempUsage>
>>>>           </systemUsage>
>>>> </systemUsage>
>>>>
>>>> So may be worth giving it a whirl if you can't scoot over to the
>>>> trunk and
>>>> ride Rob's patch.
>>>>
>>>> Joe
>>>>
>>>>
>>>> Daniel Kluesing-2 wrote:
>>>>>
>>>>> I tried the suggestion of going with the default cursor, but I
>>>>> still get
>>>>> OOM errors. I've included my full config file below, I think I'm
>>>>> running
>>>>> fairly vanilla/default.
>>>>>
>>>>> After about 350k persistent messages, the logs start to look like:
>>>>>
>>>>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index
>>>>> update took
>>>>> 3118 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 5118 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 2736 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 2945 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 33 ms, Index
>>>>> update took
>>>>> 2654 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 82 ms, Index
>>>>> update took
>>>>> 3174 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
>>>>> took
>>>>> 5891 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 2906 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 60 ms, Index
>>>>> update took
>>>>> 7619 ms
>>>>> Exception in thread "InactivityMonitor WriteCheck"
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>       at java.util.jar.Attributes.read(Attributes.java:377)
>>>>>       at java.util.jar.Manifest.read(Manifest.java:182)
>>>>>       at java.util.jar.Manifest.<init>(Manifest.java:52)
>>>>>       at
>>>>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>>>>>       at java.util.jar.JarFile.getManifest(JarFile.java:146)
>>>>>       at
>>>>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java: 
>>>>> 693)
>>>>>       at java.net.URLClassLoader.defineClass(URLClassLoader.java:
>>>>> 221)
>>>>>       at java.net.URLClassLoader.access$000(URLClassLoader.java: 
>>>>> 56)
>>>>>       at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>>>>       at java.security.AccessController.doPrivileged(Native  
>>>>> Method)
>>>>>       at java.net.URLClassLoader.findClass(URLClassLoader.java: 
>>>>> 188)
>>>>>       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>>>       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
>>>>> 319)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java: 
>>>>> 132)
>>>>>       at
>>>>> org.apache.activemq.transport.InactivityMonitor
>>>>> $2.run(InactivityMonitor.java:106)
>>>>>       at
>>>>> org
>>>>> .apache
>>>>> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java: 
>>>>> 33)
>>>>>       at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>       at java.util.TimerThread.run(Timer.java:462)
>>>>>
>>>>> Config file:
>>>>>
>>>>> <beans
>>>>> xmlns="http://www.springframework.org/schema/beans"
>>>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>>> http://activemq.apache.org/schema/core
>>>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>>>
>>>>>   <bean
>>>>> class
>>>>> =
>>>>> "org
>>>>> .springframework
>>>>> .beans.factory.config.PropertyPlaceholderConfigurer">
>>>>>       <property name="locations">
>>>>>
>>>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>>>>       </property>
>>>>>   </bean>
>>>>>
>>>>>   <broker xmlns="http://activemq.apache.org/schema/core"
>>>>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>>>>
>>>>>       <managementContext>
>>>>>           <managementContext createConnector="true"/>
>>>>>       </managementContext>
>>>>>
>>>>>       <persistenceAdapter>
>>>>>           <kahaDB directory="${activemq.base}/data/kahadb"/>
>>>>>       </persistenceAdapter>
>>>>>
>>>>> 	<destinationPolicy>
>>>>>     		<policyMap>
>>>>> 				<policyEntries>
>>>>>         			<policyEntry queue="P>" producerFlowControl="true"
>>>>> memoryLimit="10mb"></policyEntry>
>>>>>       		</policyEntries>
>>>>>     		</policyMap>
>>>>> 	</destinationPolicy>
>>>>>       <systemUsage>
>>>>>           <systemUsage sendFailIfNoSpace="true">
>>>>>           </systemUsage>
>>>>>       </systemUsage>
>>>>>       <transportConnectors>
>>>>>           <transportConnector name="openwire"
>>>>> uri="tcp://0.0.0.0:61616"/>
>>>>>       </transportConnectors>
>>>>>   </broker>
>>>>>   <import resource="jetty.xml"/>
>>>>> </beans>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>>>> Sent: Monday, January 18, 2010 10:42 PM
>>>>> To: users@activemq.apache.org
>>>>> Subject: Re: OOM with high KahaDB index time
>>>>>
>>>>>
>>>>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm running the 5.3 release as a standalone broker. In one  
>>>>>> case, a
>>>>>> producer is running without a consumer, producing small,  
>>>>>> persistent
>>>>>> messages, with the FileCursor pendingQueuePolicy (per
>>>>>> https://issues.apache.org/activemq/browse/AMQ-2512)
>>>>>> option and flow control memoryLimit set to 100mb for the queue in
>>>>>> question. (Through a policy entry)
>>>>>>
>>>>>> As the queue grows above 300k messages, KahaDB indexing starts
>>>>>> climbing above 1 second. At around 350k messages, the indexing is
>>>>>> taking over 8 seconds. At this point, I start getting java out of
>>>>>> heap space errors in essentially random parts of the code.  
>>>>>> After a
>>>>>> while, the producers timeout with a channel inactive for too long
>>>>>> error, and the entire broker basically wedges itself. At this
>>>>>> point,
>>>>>> consumers are generally unable to bind to the broker quitting  
>>>>>> with
>>>>>> timeout errors. When they can connect, consuming a single message
>>>>>> triggers an index re-build, which takes 2-8seconds. Turning on
>>>>>> verbose garbage collection, the jvm is collecting like mad but
>>>>>> reclaiming no space.
>>>>>>
>>>>>> If I restart the broker, it comes back up, I can consume the old
>>>>>> messages, and can handle another 350k messages until it wedges.
>>>>>>
>>>>>> I can reproduce under both default gc and incremental gc.
>>>>>>
>>>>>> Two questions:
>>>>>> - It seems like someone is holding onto a handle to the messages
>>>>>> after they have been persisted to disk - is this a known issue?
>>>>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>>>>
>>>>>> - Is there any documentation about the internals of KahaDB - the
>>>>>> kind of indices etc? I'd like to get a better understanding of  
>>>>>> the
>>>>>> index performance and in general how KahaDB compares to something
>>>>>> like BerkeleyDB.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> There's is some confusion over naming of our persistence options
>>>>> that
>>>>> doesn't help. There is Kaha - which uses multiple log files and a
>>>>> Hash
>>>>> based index - this is currently used by the FileCursor - whilst
>>>>> KahaDB
>>>>> is a newer implementation, which is more robust and typically  
>>>>> uses a
>>>>> BTreeIndex. There is currently a new implementation of the
>>>>> Filecursor
>>>>> btw - but that's a different matter. You can't currently configure
>>>>> the
>>>>> HashIndex via the FileCursor -  but it looks like this is the
>>>>> problem
>>>>> you are encountering - as it looks like you need to increase the  
>>>>> max
>>>>> hash buckets.
>>>>>
>>>>>
>>>>> So I would recommend the following
>>>>> 1. Use the default pendingQueuePolicy (which only uses a  
>>>>> FileCursor
>>>>> for non-persistent messages - and uses the underlying database for
>>>>> persistent messages
>>>>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>>>>> problems you are seeing with the Filecursor
>>>>>
>>>>> or - increase the maximum number of hash buckets for the  
>>>>> FileCursor
>>>>> index - by setting a Java system property -  maximumCapacity to
>>>>> 65536
>>>>> (the default is 16384)
>>>>>
>>>>> cheers,
>>>>>
>>>>> Rob
>>>>>
>>>>> http://twitter.com/rajdavies
>>>>> I work here: http://fusesource.com
>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> -- 
>>>> View this message in context:
>>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>>
>>>
>>> -- 
>>> View this message in context:
>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>>
>>
>
> -- 
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27280621.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/






Mime
View raw message