Return-Path: 
 <users-return-22167-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org>
Delivered-To: apmail-activemq-users-archive@www.apache.org
Received: (qmail 53180 invoked from network); 25 Jan 2010 15:15:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2010 15:15:24 -0000
Received: (qmail 36955 invoked by uid 500); 25 Jan 2010 15:15:24 -0000
Delivered-To: apmail-activemq-users-archive@activemq.apache.org
Received: (qmail 36899 invoked by uid 500); 25 Jan 2010 15:15:24 -0000
Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:users-help@activemq.apache.org>
List-Unsubscribe: <mailto:users-unsubscribe@activemq.apache.org>
List-Post: <mailto:users@activemq.apache.org>
List-Id: <users.activemq.apache.org>
Reply-To: users@activemq.apache.org
Delivered-To: mailing list users@activemq.apache.org
Received: (qmail 36889 invoked by uid 99); 25 Jan 2010 15:15:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 15:15:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Received: from [216.139.236.158] (HELO kuber.nabble.com) (216.139.236.158)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 15:15:13 +0000
Received: from isper.nabble.com ([192.168.236.156])
	by kuber.nabble.com with esmtp (Exim 4.63)
	(envelope-from <lists@nabble.com>)
	id 1NZQeS-000639-5B
	for users@activemq.apache.org; Mon, 25 Jan 2010 07:14:52 -0800
Message-ID: <27307957.post@talk.nabble.com>
Date: Mon, 25 Jan 2010 07:14:52 -0800 (PST)
From: Joe Fernandez <joe.fernandez@ttmsolutions.com>
To: users@activemq.apache.org
Subject: Re: OOM with high KahaDB index time
In-Reply-To: <69b2d60e1001250652q347d55b3p11ec11c3438bcbbb@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Nabble-From: joe.fernandez@ttmsolutions.com
References: 
 <33FDEB0CE2F65F41A4CF8769247BB3668D30F583A4@EXVMBX016-3.exch016.msoutlookonline.net>
 <CCBFC368-5EB9-4D85-9619-79993FA18EE5@gmail.com>
 <33FDEB0CE2F65F41A4CF8769247BB3668DB6188A23@EXVMBX016-3.exch016.msoutlookonline.net>
 <27264475.post@talk.nabble.com>
 <33FDEB0CE2F65F41A4CF8769247BB3668DB6188AE8@EXVMBX016-3.exch016.msoutlookonline.net>
 <27279301.post@talk.nabble.com>
 <69b2d60e1001250652q347d55b3p11ec11c3438bcbbb@mail.gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org


Here it is - its the xml that Dan posted with some slight mod's.  Are you
using the latest trunk and not the Jan 20 5.4 SNAPSHOT?

<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.apache.org/schema/core"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">

    <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.base}/conf/credentials.properties</value>
        </property>      
    </bean>

    <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="danbroker" dataDirectory="${activemq.base}/data">
       
        <managementContext>
            <managementContext createConnector="true"/>
        </managementContext>

        <persistenceAdapter>
            <kahaDB directory="${activemq.base}/data/kahadb"/>
        </persistenceAdapter>
   
        <destinationPolicy>
      <policyMap>
        <policyEntries>
          <policyEntry queue=">" producerFlowControl="true"
memoryLimit="10mb">
           </policyEntry>
        </policyEntries>
      </policyMap>
        </destinationPolicy>   
          

        <systemUsage>
            <systemUsage sendFailIfNoSpace="true">
            </systemUsage>
        </systemUsage> 

       
        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
        </transportConnectors>
    </broker>
    <import resource="jetty.xml"/>
</beans> 




Fred Moore-3 wrote:
> 
> Hi Joe,
> 
> can you post your whole activemq.xml for this "good enough" test on 5.3?
> 
> ...I'm still missing something and getting OOMs.
> TIA,
> F.
> 
> 
> On Fri, Jan 22, 2010 at 9:07 PM, Joe Fernandez <
> joe.fernandez@ttmsolutions.com> wrote:
> 
>>
>> I ran my 5.3 test with the following
>>
>>    <systemUsage>
>>            <systemUsage sendFailIfNoSpace="true">
>>                <memoryUsage>
>>                    <memoryUsage limit="100mb"/>
>>                </memoryUsage>
>>                <storeUsage>
>>                    <storeUsage limit="500 mb" name="foo"/>
>>                 </storeUsage>
>>                <tempUsage>
>>                    <tempUsage limit="100mb"/>
>>                </tempUsage>
>>            </systemUsage>
>>        </systemUsage>
>>
>> ...and had my producer fill up the store; it tried to pump 400k messages
>> each at 2k in size. Producer flow control was also enabled. The broker
>> did
>> not hurl any OOM exceptions, "just javax.jms.ResourceAllocationException:
>> Usage Manager Store is Full. Stopping producer..." exceptions as
>> expected.
>> The producer got kicked off its connection.
>>
>> The only time I got OOMs was when I used this.
>>
>> <systemUsage>
>>     <systemUsage sendFailIfNoSpace="true">
>>     </systemUsage>
>> </systemUsage>
>>
>> Joe
>>
>>
>>
>> Daniel Kluesing-2 wrote:
>> >
>> > So I'm not sure if that's what Rob is talking about being fixed in 5.4
>> > (And I'll try the snapshot as soon as it's ready) but if I don't have
>> the
>> > sendFailIfNoSpace then my understanding is the producers send calls
>> > block/wait/timeout - as opposed to fail - so it's more difficult to get
>> > into a HA configuration. It's a minor point, not having an OOM is much
>> > more important, but I definitely want the send calls to fail for the
>> > producer if the broker ever does anything funny.
>> >
>> > Thanks for the feedback on the config, very helpful.
>> >
>> > -----Original Message-----
>> > From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>> > Sent: Thursday, January 21, 2010 1:01 PM
>> > To: users@activemq.apache.org
>> > Subject: RE: OOM with high KahaDB index time
>> >
>> >
>> > Just for grins, I threw your cfg file into our 5.3 testbed and sure
>> > enough,
>> > we got the OOMs; I pumped 200k messages with each being 2k in size.
>> FWIW,
>> > taking this out of the cfg file made things run a lot better.
>> >
>> >  <systemUsage>
>> >      <systemUsage sendFailIfNoSpace="true">
>> >      </systemUsage>
>> > </systemUsage>
>> >
>> > With the above taken out of the cfg file, I was able to pump 400k
>> messages
>> > into the broker, no OOMs and memory utilization looked much better. I
>> also
>> > gave a fully-defined systemUsage a try and that also appeared to do the
>> > trick.
>> >
>> >  <systemUsage>
>> >             <systemUsage>
>> >                 <memoryUsage>
>> >                     <memoryUsage limit="100 mb"/>
>> >                 </memoryUsage>
>> >                 <storeUsage>
>> >                     <storeUsage limit="1 gb" name="foo"/>
>> >                 </storeUsage>
>> >                 <tempUsage>
>> >                     <tempUsage limit="100 mb"/>
>> >                 </tempUsage>
>> >             </systemUsage>
>> >  </systemUsage>
>> >
>> > So may be worth giving it a whirl if you can't scoot over to the trunk
>> and
>> > ride Rob's patch.
>> >
>> > Joe
>> >
>> >
>> > Daniel Kluesing-2 wrote:
>> >>
>> >> I tried the suggestion of going with the default cursor, but I still
>> get
>> >> OOM errors. I've included my full config file below, I think I'm
>> running
>> >> fairly vanilla/default.
>> >>
>> >> After about 350k persistent messages, the logs start to look like:
>> >>
>> >> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update
>> took
>> >> 3118 ms
>> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>> took
>> >> 5118 ms
>> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>> took
>> >> 2736 ms
>> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>> took
>> >> 2945 ms
>> >>  INFO | Slow KahaDB access: Journal append took: 33 ms, Index update
>> took
>> >> 2654 ms
>> >>  INFO | Slow KahaDB access: Journal append took: 82 ms, Index update
>> took
>> >> 3174 ms
>> >>  INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
>> took
>> >> 5891 ms
>> >>  INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>> took
>> >> 2906 ms
>> >>  INFO | Slow KahaDB access: Journal append took: 60 ms, Index update
>> took
>> >> 7619 ms
>> >> Exception in thread "InactivityMonitor WriteCheck"
>> >> java.lang.OutOfMemoryError: Java heap space
>> >>         at java.util.jar.Attributes.read(Attributes.java:377)
>> >>         at java.util.jar.Manifest.read(Manifest.java:182)
>> >>         at java.util.jar.Manifest.<init>(Manifest.java:52)
>> >>         at
>> >> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>> >>         at java.util.jar.JarFile.getManifest(JarFile.java:146)
>> >>         at
>> >> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>> >>         at
>> java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
>> >>         at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>> >>         at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>> >>         at java.security.AccessController.doPrivileged(Native Method)
>> >>         at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> >>         at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>> >>         at
>> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>> >>         at
>> >>
>> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>> >>         at
>> >>
>> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
>> >>         at
>> >>
>> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>> >>         at java.util.TimerThread.mainLoop(Timer.java:512)
>> >>         at java.util.TimerThread.run(Timer.java:462)
>> >>
>> >> Config file:
>> >>
>> >> <beans
>> >>   xmlns="http://www.springframework.org/schema/beans"
>> >>   xmlns:amq="http://activemq.apache.org/schema/core"
>> >>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>   xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>> >>   http://activemq.apache.org/schema/core
>> >> http://activemq.apache.org/schema/core/activemq-core.xsd">
>> >>
>> >>     <bean
>> >>
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>> >>         <property name="locations">
>> >>
>> >> <value>file:${activemq.base}/conf/credentials.properties</value>
>> >>         </property>
>> >>     </bean>
>> >>
>> >>     <broker xmlns="http://activemq.apache.org/schema/core"
>> >> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>> >>
>> >>         <managementContext>
>> >>             <managementContext createConnector="true"/>
>> >>         </managementContext>
>> >>
>> >>         <persistenceAdapter>
>> >>             <kahaDB directory="${activemq.base}/data/kahadb"/>
>> >>         </persistenceAdapter>
>> >>
>> >>      <destinationPolicy>
>> >>                      <policyMap>
>> >>                              <policyEntries>
>> >>                              <policyEntry queue="P>"
>> producerFlowControl="true"
>> >> memoryLimit="10mb"></policyEntry>
>> >>                      </policyEntries>
>> >>                      </policyMap>
>> >>      </destinationPolicy>
>> >>         <systemUsage>
>> >>             <systemUsage sendFailIfNoSpace="true">
>> >>             </systemUsage>
>> >>         </systemUsage>
>> >>         <transportConnectors>
>> >>             <transportConnector name="openwire"
>> >> uri="tcp://0.0.0.0:61616"/>
>> >>         </transportConnectors>
>> >>     </broker>
>> >>     <import resource="jetty.xml"/>
>> >> </beans>
>> >>
>> >> -----Original Message-----
>> >> From: Rob Davies [mailto:rajdavies@gmail.com]
>> >> Sent: Monday, January 18, 2010 10:42 PM
>> >> To: users@activemq.apache.org
>> >> Subject: Re: OOM with high KahaDB index time
>> >>
>> >>
>> >> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I'm running the 5.3 release as a standalone broker. In one case, a
>> >>> producer is running without a consumer, producing small, persistent
>> >>> messages, with the FileCursor pendingQueuePolicy (per
>> >>> https://issues.apache.org/activemq/browse/AMQ-2512)
>> >>>  option and flow control memoryLimit set to 100mb for the queue in
>> >>> question. (Through a policy entry)
>> >>>
>> >>> As the queue grows above 300k messages, KahaDB indexing starts
>> >>> climbing above 1 second. At around 350k messages, the indexing is
>> >>> taking over 8 seconds. At this point, I start getting java out of
>> >>> heap space errors in essentially random parts of the code. After a
>> >>> while, the producers timeout with a channel inactive for too long
>> >>> error, and the entire broker basically wedges itself. At this point,
>> >>> consumers are generally unable to bind to the broker quitting with
>> >>> timeout errors. When they can connect, consuming a single message
>> >>> triggers an index re-build, which takes 2-8seconds. Turning on
>> >>> verbose garbage collection, the jvm is collecting like mad but
>> >>> reclaiming no space.
>> >>>
>> >>> If I restart the broker, it comes back up, I can consume the old
>> >>> messages, and can handle another 350k messages until it wedges.
>> >>>
>> >>> I can reproduce under both default gc and incremental gc.
>> >>>
>> >>> Two questions:
>> >>> - It seems like someone is holding onto a handle to the messages
>> >>> after they have been persisted to disk - is this a known issue?
>> >>> Should I open a JIRA for it? (Or is there another explanation?)
>> >>>
>> >>> - Is there any documentation about the internals of KahaDB - the
>> >>> kind of indices etc? I'd like to get a better understanding of the
>> >>> index performance and in general how KahaDB compares to something
>> >>> like BerkeleyDB.
>> >>>
>> >>> Thanks
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >> There's is some confusion over naming of our persistence options that
>> >> doesn't help. There is Kaha - which uses multiple log files and a Hash
>> >> based index - this is currently used by the FileCursor - whilst KahaDB
>> >> is a newer implementation, which is more robust and typically uses a
>> >> BTreeIndex. There is currently a new implementation of the Filecursor
>> >> btw - but that's a different matter. You can't currently configure the
>> >> HashIndex via the FileCursor -  but it looks like this is the problem
>> >> you are encountering - as it looks like you need to increase the max
>> >> hash buckets.
>> >>
>> >>
>> >> So I would recommend the following
>> >> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
>> >> for non-persistent messages - and uses the underlying database for
>> >> persistent messages
>> >> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>> >> problems you are seeing with the Filecursor
>> >>
>> >> or - increase the maximum number of hash buckets for the FileCursor
>> >> index - by setting a Java system property -  maximumCapacity to 65536
>> >> (the default is 16384)
>> >>
>> >> cheers,
>> >>
>> >> Rob
>> >>
>> >> http://twitter.com/rajdavies
>> >> I work here: http://fusesource.com
>> >> My Blog: http://rajdavies.blogspot.com/
>> >> I'm writing this: http://www.manning.com/snyder/
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >
>> > --
>> > View this message in context:
>> >
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27307957.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.