activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: java/lang/OutOfMemoryError
Date Wed, 14 Oct 2009 09:00:09 GMT
the config looks ok,  just one thought, do you need multicast discovery on
your transport connector? I wonder if it is possible you are seeing
https://issues.apache.org/activemq/browse/AMQ-2283 which resulted in an leak
of discovery threads? Removing the discoveryUri will work around it.

2009/10/13 tejeswara <tejeswara@gmail.com>

>
> Here is our activemq.xml. As you can see it is pretty much same as the
> template version we get from activemq package:
> <beans
>  xmlns="http://www.springframework.org/schema/beans"
>  xmlns:amq="http://activemq.apache.org/schema/core"
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>  xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>  http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd
>  http://activemq.apache.org/camel/schema/spring
> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
>    <bean
>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>         <property name="locations">
>
> <value>file:///${activemq.base}/conf/credentials.properties</value>
>         </property>
>    </bean>
>    <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.base}/data">
>        <destinationPolicy>
>            <policyMap>
>                <policyEntries>
>                    <policyEntry queue=">" memoryLimit="50mb"/>
>                    <policyEntry topic=">" memoryLimit="5mb">
>                    </policyEntry>
>                </policyEntries>
>            </policyMap>
>        </destinationPolicy>
>        <managementContext>
>            <managementContext createConnector="true"
> connectorPort="11099"/>
>        </managementContext>
>        <networkConnectors>
>            <networkConnector name="default-nc" uri="multicast://default"/>
>        </networkConnectors>
>        <persistenceAdapter>
>            <amqPersistenceAdapter syncOnWrite="false"
> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>        </persistenceAdapter>
>        <systemUsage>
>            <systemUsage>
>                <memoryUsage>
>                    <memoryUsage limit="200 mb"/>
>                </memoryUsage>
>                <storeUsage>
>                    <storeUsage limit="1 gb" name="foo"/>
>                </storeUsage>
>                <tempUsage>
>                    <tempUsage limit="100 mb"/>
>                </tempUsage>
>            </systemUsage>
>        </systemUsage>
>        <transportConnectors>
>            <transportConnector name="openwire" uri="tcp://localhost:61616"
> discoveryUri="multicast://default"/>
>        </transportConnectors>
>    </broker>
>    <camelContext id="camel"
> xmlns="http://activemq.apache.org/camel/schema/spring">
>        <package>org.foo.bar</package>
>        <route>
>            <from uri="activemq:example.A"/>
>            <to uri="activemq:example.B"/>
>        </route>
>    </camelContext>
>    <bean id="activemq"
> class="org.apache.activemq.camel.component.ActiveMQComponent" >
>        <property name="connectionFactory">
>          <bean class="org.apache.activemq.ActiveMQConnectionFactory">
>            <property name="brokerURL"
> value="vm://localhost?create=false&amp;waitForStart=10000" />
>            <property name="userName" value="${activemq.username}"/>
>            <property name="password" value="${activemq.password}"/>
>          </bean>
>        </property>
>    </bean>
>    <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
>        <connectors>
>            <nioConnector port="11098"/>
>        </connectors>
>        <handlers>
>            <webAppContext contextPath="/admin"
> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>            <webAppContext contextPath="/demo"
> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
>            <webAppContext contextPath="/fileserver"
> resourceBase="${activemq.base}/webapps/fileserver" logUrlOnStart="true"/>
>        </handlers>
>    </jetty>
> </beans>
>
>
>
> Gary Tully wrote:
> >
> > Can you post your activemq.xml? It looks like you need to restrict the
> > memory usage of the broker and have it spool messages to disk when the
> > memory limits are reached. See some additional detail at
> >
> http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk
> > *
> >
> > *
> > 2009/10/12 tejeswara <tejeswara@gmail.com>
> >
> >>
> >> Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I see
> >> below
> >> error in logs.
> >> Note that, below values are set in bin/activemq in out test lab:
> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
> >>  ACTIVEMQ_OPTS="-Xmx512M
> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=true"
> >> fi
> >>
> >> I tried to stress test broker by creating 200 queues and put 5000
> >> messages
> >> (each about 1500 bytes size)
> >> in each of these queues. However I could not see any error.  Below error
> >> occured when broker is idle.
> >> From logs, only activity during this idle period I see is 'DEBUG
> >> InactivityMonitor              - 30177 ms elapsed since last read check'
> >>
> >> Can someone help in pointing out what is going wrong?
> >>
> >> Note that, OutOfMemoryError is seen in our production even when heap
> size
> >> is
> >> bumped up to  1024 and
> >> dedicated task runner is set to false. This error usvally occurs after 2
> >> days broker running without restart.
> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then
> >>  ACTIVEMQ_OPTS="-Xmx1024M
> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
> >> fi
> >>
> >>
> >> Stack Trace:->
> >>
> >> JVMDUMP006I Processing Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError" - Please Wait.
> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc'
> >> JVMDUMP010I Snap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc
> >> JVMDUMP007I JVM Requesting Heap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd'
> >> JVMDUMP010I Heap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd
> >> JVMDUMP007I JVM Requesting Java Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt'
> >> JVMDUMP010I Java Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt
> >> JVMDUMP013I Processed Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError".
> >> Exception in thread "MulticastDiscovery: null"
> >> java.lang.OutOfMemoryError:
> >> heap allocation failed
> >>        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
> >>        at
> >>
> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
> >>        at java.net.DatagramSocket.receive(DatagramSocket.java:724)
> >>        at
> >>
> >>
> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
> >>        at java.lang.Thread.run(Thread.java:803)
> >> JVMDUMP006I Processing Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError" - Please Wait.
> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc'
> >> JVMDUMP010I Snap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc
> >> JVMDUMP007I JVM Requesting Heap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd'
> >> JVMDUMP016E Aborting: Cannot create file
> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd)
> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
> >> JVMDUMP007I JVM Requesting Java Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt'
> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
> >> JVMDUMP010I Java Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt
> >> JVMDUMP013I Processed Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError".
> >> Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616"
> >> java.lang.OutOfMemoryError: heap allocation failed
> >>        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
> >>        at
> >>
> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181)
> >>        at java.net.DatagramSocket.receive(DatagramSocket.java:724)
> >>        at
> >>
> >>
> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338)
> >>        at java.lang.Thread.run(Thread.java:803)
> >> JVMDUMP006I Processing Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError" - Please Wait.
> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc'
> >> JVMDUMP010I Snap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc
> >> JVMDUMP007I JVM Requesting Heap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd'
> >> JVMDUMP016E Aborting: Cannot create file
> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd)
> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written
> >> JVMDUMP007I JVM Requesting Java Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt'
> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer
> >> JVMDUMP010I Java Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt
> >> JVMDUMP013I Processed Dump Event "uncaught", detail
> >> "java/lang/OutOfMemoryError".
> >> Exception in thread "InactivityMonitor WriteCheck"
> >> java.lang.OutOfMemoryError
> >>        at java.security.AccessController.getProtectionDomains(Native
> >> Method)
> >>        at
> >> java.security.AccessController.getContext(AccessController.java:132)
> >>        at java.lang.Thread.initialize(Thread.java:347)
> >>        at java.lang.Thread.<init>(Thread.java:278)
> >>        at java.lang.Thread.<init>(Thread.java:170)
> >>        at
> >>
> >>
> org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303)
> >>        at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411)
> >>        at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463)
> >>        at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878)
> >>        at
> >>
> >>
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
> >>        at
> >>
> >>
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
> >>        at
> >>
> >>
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> >>        at java.util.TimerThread.mainLoop(Timer.java:527)
> >>  at java.util.TimerThread.run(Timer.java:477)
> >> Exception in thread "RMI TCP Connection(454)-10.15.26.210"
> >> java.lang.NoClassDefFoundError: java.io.StreamCorruptedException
> >>        at
> >>
> >>
> sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177)
> >>        at
> >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335)
> >>        at sun.rmi.transport.Transport$1.run(Transport.java:168)
> >>        at
> >> java.security.AccessController.doPrivileged(AccessController.java:275)
> >> java.lang.NoClassDefFoundError: java.rmi.UnmarshalException
> >>        at
> >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
> >>        at
> >>
> >>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
> >>        at
> >>
> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
> >>        at
> >>
> >>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
> >>        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358)
> >>        at java.lang.Thread.run(Thread.java:803)
> >>        at sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67)
> >>        at
> >>
> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344)
> >>        at
> >>
> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160)
> >>        at
> >>
> >>
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
> >>        at java.lang.Thread.run(Thread.java:803)
> >> Exception in thread "RMI LeaseChecker" java.lang.NoClassDefFoundError:
> >> java.rmi.server.Unreferenced
> >>        at sun.rmi.transport.Target.refSetRemove(Target.java:320)
> >>        at sun.rmi.transport.Target.vmidDead(Target.java:444)
> >>        at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265)
> >>        at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47)
> >>        at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329)
> >>        at java.lang.Thread.run(Thread.java:803)
> >> Unhandled exception
> >> Type=Segmentation error vmState=0x00053fff
> >> J9Generic_Signal_Number=00000004 Signal_Number=0000000b
> >> Error_Value=00000000
> >> Signal_Code=00000001
> >> Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000
> >> EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000
> >> ECX=08B7DF40 EDX=00000000
> >> EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8
> >> EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8
> >> Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so
> >> Module_base_address=00F09000
> >>
> >>
> >>
> Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V
> >> Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp)
> >> CPU=x86 (8 logical CPUs) (0x47621d000 RAM)
> >> JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
> >> JVMDUMP007I JVM Requesting System Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp'
> >> JVMDUMP012E Error in System Dump: cannot find core file, check "ulimit
> >> -c"
> >> is set high enough
> >> JVMDUMP007I JVM Requesting Snap Dump using
> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc'
> >> JVMDUMP010I Snap Dump written to
> >> /x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc
> >> JVMDUMP007I JVM Requesting Java Dump using
> >> '/tmp/javacore.20091011.220932.16581.txt'
> >> NULL
> >> ------------------------------------------------------------------------
> >> 0SECTION       TITLE subcomponent dump routine
> >> NULL           ===============================
> >> 1TISIGINFO     Dump Event "gpf" (00002000) received
> >> 1TIDATETIME    Date:                 2009/10/11 at 22:10:17
> >> 1TIFILENAME    Javacore filename:
> >>  /tmp/javacore.20091011.220932.16581.txt
> >> NULL
> >> ------------------------------------------------------------------------
> >> 0SECTION       GPINFO subcomponent dump routine
> >> NULL           ================================
> >> 2XHOSLEVEL     OS Level         : Linux 2.6.9-55.ELsmp
> >> 2XHCPUS        Processors -
> >> 3XHCPUARCH       Architecture   : x86
> >> 3XHNUMCPUS       How Many       : 8
> >> NULL
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25881255.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message