activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Justin C. van Vorst" <pra...@gmail.com>
Subject OutOfMemoryError using 4.1.1
Date Mon, 23 Apr 2007 16:01:37 GMT

On our production server this weekend, activemq-4.1.1 hit a
java.lang.OutOfMemoryError exception and then proceeded to stop functioning. 
The process was using about 3GB of RAM.

I've seen some other posts about OOME, but none that quite match the stack
of ours.  Does anyone have a place for me to start investigating this issue?

Below are the error logs, command line start-up, and activemq.xml config:


Error log:

2007-04-23 02:43:10,286 [localhost:61613] WARN  ManagedTransportConnection    
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_54813
2007-04-23 02:49:10,415 [localhost:61613] WARN  ManagedTransportConnection    
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_58365
2007-04-23 03:11:10,301 [localhost:61613] WARN  ManagedTransportConnection    
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_42109
2007-04-23 03:15:10,056 [localhost:61613] WARN  ManagedTransportConnection    
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_44595
2007-04-23 03:27:10,120 [localhost:61613] WARN  ManagedTransportConnection    
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51338
2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter     
- Failed to checkpoint a message store:
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: unable to create new native thread
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: unable to create new native thread
        at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299)
        at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
        at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
        at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
        at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:574)
        at java.util.Timer.<init>(Timer.java:154)
        at java.util.Timer.<init>(Timer.java:122)
        at com.mysql.jdbc.Connection.<init>(Connection.java:1441)
        at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
        at
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
        at
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
        at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771)
        at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
        at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544)
        at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:55)
        at
org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:147)
        at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189)
        at
org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41)
        at
org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247)
        at
org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356)
        at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
        at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)
2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter     
- Failed to mark the Journal:
org.apache.activeio.journal.InvalidRecordLocationException: The location is
less than the last mark.
org.apache.activeio.journal.InvalidRecordLocationException: The location is
less than the last mark.
        at
org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
        at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
        at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
        at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
2007-04-23 03:47:10,105 [localhost:61613] WARN  ManagedTransportConnection    
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_34348
2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter     
- Failed to checkpoint a message store:
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.io.IOException: Already started.
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.io.IOException: Already started.
        at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299)
        at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
        at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
        at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
        at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
Caused by: java.io.IOException: Already started.
        at
org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:145)
        at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189)
        at
org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41)
        at
org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247)
        at
org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356)
        at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
        at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)
2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter     
- Failed to mark the Journal:
org.apache.activeio.journal.InvalidRecordLocationException: The location is
less than the last mark.
org.apache.activeio.journal.InvalidRecordLocationException: The location is
less than the last mark.
        at
org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403)
        at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
        at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
        at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
        at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
2007-04-23 03:51:10,188 [localhost:61613] WARN  ManagedTransportConnection    
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_36436
2007-04-23 04:13:10,134 [localhost:61613] WARN  ManagedTransportConnection    
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_49022
2007-04-23 04:17:10,094 [localhost:61613] WARN  ManagedTransportConnection    
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51068



Command line:

usr/java/current/bin/java -Xmx512M
-Dorg.apache.activemq.UseDedicatedTaskRunner=true
-Dderby.system.home=/opt/activemq-4.1.1/data
-Dderby.storage.fileSyncTransactionLog=true -Dcom.sun.management.jmxremote
-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.ssl.trustStorePassword=password
-Djavax.net.ssl.keyStore=/opt/activemq-4.1.1/conf/broker.ks
-Djavax.net.ssl.trustStore=/opt/activemq-4.1.1/conf/broker.ts -classpath 
-Dactivemq.home=/opt/rhg/activemq-4.1.1 -Dactivemq.base=/opt/activemq-4.1.1
-jar /opt/activemq-4.1.1/bin/run.jar



activemq.xml:

<beans>

  <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>

  <broker brokerName="localhost" useJmx="true"
xmlns="http://activemq.org/config/1.0">

    <destinationPolicy>
      <policyMap><policyEntries>
          <policyEntry topic="FOO.>">
            <dispatchPolicy>
              <strictOrderDispatchPolicy />
            </dispatchPolicy>
            <subscriptionRecoveryPolicy>
              <lastImageSubscriptionRecoveryPolicy />
            </subscriptionRecoveryPolicy>
          </policyEntry>
      </policyEntries></policyMap>
    </destinationPolicy>

    <persistenceAdapter>
      <journaledJDBC journalLogFiles="5" dataDirectory="../activemq-data"
dataSource="#mysql-ds"/>
    </persistenceAdapter>

    <transportConnectors>
       <transportConnector name="openwire" uri="tcp://localhost:61616"
discoveryUri="multicast://default"/>
       <transportConnector name="ssl"     uri="ssl://localhost:61617"/>
       <transportConnector name="stomp"   uri="stomp://localhost:61613"/>
    </transportConnectors>

    <networkConnectors>
      <networkConnector name="default-nc" uri="multicast://default"/>
    </networkConnectors>

  </broker>

  <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
    <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
    <property name="url"
value="jdbc:mysql://x.x.x.x/activemq?relaxAutoCommit=true"/>
    <property name="username" value="username"/>
    <property name="password" value="password"/>
    <property name="poolPreparedStatements" value="true"/>
  </bean>

</beans>
-- 
View this message in context: http://www.nabble.com/OutOfMemoryError-using-4.1.1-tf3632613s2354.html#a10143322
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message