activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: OutOfMemoryError using 4.1.1
Date Tue, 24 Apr 2007 16:58:20 GMT
It might be you are just running out of threads; maybe your setting of
file descriptors per process is too low?


On 4/23/07, Justin C. van Vorst <praxeo@gmail.com> wrote:
>
> On our production server this weekend, activemq-4.1.1 hit a
> java.lang.OutOfMemoryError exception and then proceeded to stop functioning.
> The process was using about 3GB of RAM.
>
> I've seen some other posts about OOME, but none that quite match the stack
> of ours.  Does anyone have a place for me to start investigating this issue?
>
> Below are the error logs, command line start-up, and activemq.xml config:
>
>
> Error log:
>
> 2007-04-23 02:43:10,286 [localhost:61613] WARN  ManagedTransportConnection
> - Failed to register MBean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_54813
> 2007-04-23 02:49:10,415 [localhost:61613] WARN  ManagedTransportConnection
> - Failed to register MBean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_58365
> 2007-04-23 03:11:10,301 [localhost:61613] WARN  ManagedTransportConnection
> - Failed to register MBean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_42109
> 2007-04-23 03:15:10,056 [localhost:61613] WARN  ManagedTransportConnection
> - Failed to register MBean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_44595
> 2007-04-23 03:27:10,120 [localhost:61613] WARN  ManagedTransportConnection
> - Failed to register MBean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51338
> 2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter
> - Failed to checkpoint a message store:
> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
> java.lang.OutOfMemoryError: unable to create new native thread
> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
> java.lang.OutOfMemoryError: unable to create new native thread
>         at
> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299)
>         at
> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:574)
>         at java.util.Timer.<init>(Timer.java:154)
>         at java.util.Timer.<init>(Timer.java:122)
>         at com.mysql.jdbc.Connection.<init>(Connection.java:1441)
>         at
> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
>         at
> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
>         at
> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
>         at
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771)
>         at
> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
>         at
> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544)
>         at
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:55)
>         at
> org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:147)
>         at
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189)
>         at
> org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41)
>         at
> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247)
>         at
> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356)
>         at
> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
>         at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>         at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>         at java.lang.Thread.run(Thread.java:595)
> 2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter
> - Failed to mark the Journal:
> org.apache.activeio.journal.InvalidRecordLocationException: The location is
> less than the last mark.
> org.apache.activeio.journal.InvalidRecordLocationException: The location is
> less than the last mark.
>         at
> org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
> 2007-04-23 03:47:10,105 [localhost:61613] WARN  ManagedTransportConnection
> - Failed to register MBean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_34348
> 2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter
> - Failed to checkpoint a message store:
> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
> java.io.IOException: Already started.
> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
> java.io.IOException: Already started.
>         at
> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299)
>         at
> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
> Caused by: java.io.IOException: Already started.
>         at
> org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:145)
>         at
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189)
>         at
> org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41)
>         at
> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247)
>         at
> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356)
>         at
> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
>         at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>         at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>         at java.lang.Thread.run(Thread.java:595)
> 2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter
> - Failed to mark the Journal:
> org.apache.activeio.journal.InvalidRecordLocationException: The location is
> less than the last mark.
> org.apache.activeio.journal.InvalidRecordLocationException: The location is
> less than the last mark.
>         at
> org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403)
>         at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>         at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
> 2007-04-23 03:51:10,188 [localhost:61613] WARN  ManagedTransportConnection
> - Failed to register MBean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_36436
> 2007-04-23 04:13:10,134 [localhost:61613] WARN  ManagedTransportConnection
> - Failed to register MBean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_49022
> 2007-04-23 04:17:10,094 [localhost:61613] WARN  ManagedTransportConnection
> - Failed to register MBean:
> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51068
>
>
>
> Command line:
>
> usr/java/current/bin/java -Xmx512M
> -Dorg.apache.activemq.UseDedicatedTaskRunner=true
> -Dderby.system.home=/opt/activemq-4.1.1/data
> -Dderby.storage.fileSyncTransactionLog=true -Dcom.sun.management.jmxremote
> -Djavax.net.ssl.keyStorePassword=password
> -Djavax.net.ssl.trustStorePassword=password
> -Djavax.net.ssl.keyStore=/opt/activemq-4.1.1/conf/broker.ks
> -Djavax.net.ssl.trustStore=/opt/activemq-4.1.1/conf/broker.ts -classpath
> -Dactivemq.home=/opt/rhg/activemq-4.1.1 -Dactivemq.base=/opt/activemq-4.1.1
> -jar /opt/activemq-4.1.1/bin/run.jar
>
>
>
> activemq.xml:
>
> <beans>
>
>   <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
>
>   <broker brokerName="localhost" useJmx="true"
> xmlns="http://activemq.org/config/1.0">
>
>     <destinationPolicy>
>       <policyMap><policyEntries>
>           <policyEntry topic="FOO.>">
>             <dispatchPolicy>
>               <strictOrderDispatchPolicy />
>             </dispatchPolicy>
>             <subscriptionRecoveryPolicy>
>               <lastImageSubscriptionRecoveryPolicy />
>             </subscriptionRecoveryPolicy>
>           </policyEntry>
>       </policyEntries></policyMap>
>     </destinationPolicy>
>
>     <persistenceAdapter>
>       <journaledJDBC journalLogFiles="5" dataDirectory="../activemq-data"
> dataSource="#mysql-ds"/>
>     </persistenceAdapter>
>
>     <transportConnectors>
>        <transportConnector name="openwire" uri="tcp://localhost:61616"
> discoveryUri="multicast://default"/>
>        <transportConnector name="ssl"     uri="ssl://localhost:61617"/>
>        <transportConnector name="stomp"   uri="stomp://localhost:61613"/>
>     </transportConnectors>
>
>     <networkConnectors>
>       <networkConnector name="default-nc" uri="multicast://default"/>
>     </networkConnectors>
>
>   </broker>
>
>   <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
>     <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
>     <property name="url"
> value="jdbc:mysql://x.x.x.x/activemq?relaxAutoCommit=true"/>
>     <property name="username" value="username"/>
>     <property name="password" value="password"/>
>     <property name="poolPreparedStatements" value="true"/>
>   </bean>
>
> </beans>
> --
> View this message in context: http://www.nabble.com/OutOfMemoryError-using-4.1.1-tf3632613s2354.html#a10143322
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
James
-------
http://macstrac.blogspot.com/

Mime
View raw message