activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Justin C. van Vorst" <pra...@gmail.com>
Subject Re: OutOfMemoryError using 4.1.1
Date Mon, 07 May 2007 15:46:22 GMT

What is the correct number of file descriptors necessary to not have ActiveMQ
fail in a production environment?



James.Strachan wrote:
> 
> It might be you are just running out of threads; maybe your setting of
> file descriptors per process is too low?
> 
> 
> On 4/23/07, Justin C. van Vorst <praxeo@gmail.com> wrote:
>>
>> On our production server this weekend, activemq-4.1.1 hit a
>> java.lang.OutOfMemoryError exception and then proceeded to stop
>> functioning.
>> The process was using about 3GB of RAM.
>>
>> I've seen some other posts about OOME, but none that quite match the
>> stack
>> of ours.  Does anyone have a place for me to start investigating this
>> issue?
>>
>> Below are the error logs, command line start-up, and activemq.xml config:
>>
>>
>> Error log:
>>
>> 2007-04-23 02:43:10,286 [localhost:61613] WARN 
>> ManagedTransportConnection
>> - Failed to register MBean:
>> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_54813
>> 2007-04-23 02:49:10,415 [localhost:61613] WARN 
>> ManagedTransportConnection
>> - Failed to register MBean:
>> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_58365
>> 2007-04-23 03:11:10,301 [localhost:61613] WARN 
>> ManagedTransportConnection
>> - Failed to register MBean:
>> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_42109
>> 2007-04-23 03:15:10,056 [localhost:61613] WARN 
>> ManagedTransportConnection
>> - Failed to register MBean:
>> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_44595
>> 2007-04-23 03:27:10,120 [localhost:61613] WARN 
>> ManagedTransportConnection
>> - Failed to register MBean:
>> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51338
>> 2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter
>> - Failed to checkpoint a message store:
>> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
>> java.lang.OutOfMemoryError: unable to create new native thread
>> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
>> java.lang.OutOfMemoryError: unable to create new native thread
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299)
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>         at java.lang.Thread.start0(Native Method)
>>         at java.lang.Thread.start(Thread.java:574)
>>         at java.util.Timer.<init>(Timer.java:154)
>>         at java.util.Timer.<init>(Timer.java:122)
>>         at com.mysql.jdbc.Connection.<init>(Connection.java:1441)
>>         at
>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
>>         at
>> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
>>         at
>> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
>>         at
>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771)
>>         at
>> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
>>         at
>> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544)
>>         at
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:55)
>>         at
>> org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:147)
>>         at
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189)
>>         at
>> org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41)
>>         at
>> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247)
>>         at
>> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356)
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>>         at java.lang.Thread.run(Thread.java:595)
>> 2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter
>> - Failed to mark the Journal:
>> org.apache.activeio.journal.InvalidRecordLocationException: The location
>> is
>> less than the last mark.
>> org.apache.activeio.journal.InvalidRecordLocationException: The location
>> is
>> less than the last mark.
>>         at
>> org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
>> 2007-04-23 03:47:10,105 [localhost:61613] WARN 
>> ManagedTransportConnection
>> - Failed to register MBean:
>> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_34348
>> 2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter
>> - Failed to checkpoint a message store:
>> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
>> java.io.IOException: Already started.
>> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
>> java.io.IOException: Already started.
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299)
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
>> Caused by: java.io.IOException: Already started.
>>         at
>> org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:145)
>>         at
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189)
>>         at
>> org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41)
>>         at
>> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247)
>>         at
>> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356)
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>>         at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>>         at java.lang.Thread.run(Thread.java:595)
>> 2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter
>> - Failed to mark the Journal:
>> org.apache.activeio.journal.InvalidRecordLocationException: The location
>> is
>> less than the last mark.
>> org.apache.activeio.journal.InvalidRecordLocationException: The location
>> is
>> less than the last mark.
>>         at
>> org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403)
>>         at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>>         at
>> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
>> 2007-04-23 03:51:10,188 [localhost:61613] WARN 
>> ManagedTransportConnection
>> - Failed to register MBean:
>> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_36436
>> 2007-04-23 04:13:10,134 [localhost:61613] WARN 
>> ManagedTransportConnection
>> - Failed to register MBean:
>> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_49022
>> 2007-04-23 04:17:10,094 [localhost:61613] WARN 
>> ManagedTransportConnection
>> - Failed to register MBean:
>> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51068
>>
>>
>>
>> Command line:
>>
>> usr/java/current/bin/java -Xmx512M
>> -Dorg.apache.activemq.UseDedicatedTaskRunner=true
>> -Dderby.system.home=/opt/activemq-4.1.1/data
>> -Dderby.storage.fileSyncTransactionLog=true
>> -Dcom.sun.management.jmxremote
>> -Djavax.net.ssl.keyStorePassword=password
>> -Djavax.net.ssl.trustStorePassword=password
>> -Djavax.net.ssl.keyStore=/opt/activemq-4.1.1/conf/broker.ks
>> -Djavax.net.ssl.trustStore=/opt/activemq-4.1.1/conf/broker.ts -classpath
>> -Dactivemq.home=/opt/rhg/activemq-4.1.1
>> -Dactivemq.base=/opt/activemq-4.1.1
>> -jar /opt/activemq-4.1.1/bin/run.jar
>>
>>
>>
>> activemq.xml:
>>
>> <beans>
>>
>>   <bean
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
>>
>>   <broker brokerName="localhost" useJmx="true"
>> xmlns="http://activemq.org/config/1.0">
>>
>>     <destinationPolicy>
>>       <policyMap><policyEntries>
>>           <policyEntry topic="FOO.>">
>>             <dispatchPolicy>
>>               <strictOrderDispatchPolicy />
>>             </dispatchPolicy>
>>             <subscriptionRecoveryPolicy>
>>               <lastImageSubscriptionRecoveryPolicy />
>>             </subscriptionRecoveryPolicy>
>>           </policyEntry>
>>       </policyEntries></policyMap>
>>     </destinationPolicy>
>>
>>     <persistenceAdapter>
>>       <journaledJDBC journalLogFiles="5" dataDirectory="../activemq-data"
>> dataSource="#mysql-ds"/>
>>     </persistenceAdapter>
>>
>>     <transportConnectors>
>>        <transportConnector name="openwire" uri="tcp://localhost:61616"
>> discoveryUri="multicast://default"/>
>>        <transportConnector name="ssl"     uri="ssl://localhost:61617"/>
>>        <transportConnector name="stomp"   uri="stomp://localhost:61613"/>
>>     </transportConnectors>
>>
>>     <networkConnectors>
>>       <networkConnector name="default-nc" uri="multicast://default"/>
>>     </networkConnectors>
>>
>>   </broker>
>>
>>   <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
>> destroy-method="close">
>>     <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
>>     <property name="url"
>> value="jdbc:mysql://x.x.x.x/activemq?relaxAutoCommit=true"/>
>>     <property name="username" value="username"/>
>>     <property name="password" value="password"/>
>>     <property name="poolPreparedStatements" value="true"/>
>>   </bean>
>>
>> </beans>
>> --
>> View this message in context:
>> http://www.nabble.com/OutOfMemoryError-using-4.1.1-tf3632613s2354.html#a10143322
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> 

-- 
View this message in context: http://www.nabble.com/OutOfMemoryError-using-4.1.1-tf3632613s2354.html#a10359823
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message