activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cott Lang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMQ-6451) Preallocation strategy of zeros can leak file descriptors causing crash
Date Mon, 03 Oct 2016 17:09:20 GMT
Cott Lang created AMQ-6451:
------------------------------

             Summary: Preallocation strategy of zeros can leak file descriptors causing crash
                 Key: AMQ-6451
                 URL: https://issues.apache.org/jira/browse/AMQ-6451
             Project: ActiveMQ
          Issue Type: Bug
          Components: KahaDB
    Affects Versions: 5.14.0
            Reporter: Cott Lang
            Priority: Critical


If the preallocationStrategy is set to 'zeros', ActiveMQ can intermittently become unable
to allocate direct buffer memory with the default JVM settings. The exception isn't handled,
and ends up both creating an empty journal file and, more importantly, leaking a file descriptor.

ActiveMQ eventually runs out of file descriptors and crashes.

In addition to handling this condition, perhaps the default ACTIVEMQ_OPTS_MEMORY settings
should configure enough direct memory to allow some multiple of log files to be created near
simultaneously, or at least this possibility documented in the KahaDB settings.

Relevant logs:

{noformat}
2016-10-03 12:47:24,154 | WARN  | Async error occurred: java.lang.OutOfMemoryError: Direct
buffer memory | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
ssl:///x.x.x.x:60805
2016-10-03 12:47:24,818 | WARN  | Async error occurred: java.lang.OutOfMemoryError: Direct
buffer memory | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
ssl:///x.x.x.x:60811
2016-10-03 12:47:25,477 | WARN  | Async error occurred: java.lang.OutOfMemoryError: Direct
buffer memory | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
ssl:///x.x.x.x:49830
2016-10-03 12:47:26,146 | WARN  | Store COMMIT FAILED:  | org.apache.activemq.transaction.LocalTransaction
| ActiveMQ Transport: ssl:///x.x.x.x:65534
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:693)[:1.8.0_101]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)[:1.8.0_101]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)[:1.8.0_101]
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)[:1.8.0_101]
        at sun.nio.ch.IOUtil.write(IOUtil.java:58)[:1.8.0_101]
        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)[:1.8.0_101]
        at org.apache.activemq.store.kahadb.disk.journal.Journal.doPreallocationZeros(Journal.java:366)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.disk.journal.Journal.preallocateEntireJournalDataFile(Journal.java:333)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.disk.journal.Journal.newDataFile(Journal.java:631)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.disk.journal.Journal.rotateWriteFile(Journal.java:595)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.disk.journal.Journal.getCurrentDataFile(Journal.java:984)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.enqueue(DataFileAppender.java:189)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:128)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:890)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1108)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1090)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.store.kahadb.KahaDBTransactionStore.commit(KahaDBTransactionStore.java:301)[activemq-kahadb-store-5.14.0.jar:5.14.0]
        at org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:70)[activemq-broker-5.14.0.jar:5.14.0]
        at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:252)[activemq-broker-5.14.0.jar:5.14.0]
        at org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:113)[activemq-broker-5.14.0.jar:5.14.0]
        at org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:113)[activemq-broker-5.14.0.jar:5.14.0]
        at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:118)[activemq-broker-5.14.0.jar:5.14.0]
        at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:535)[activemq-broker-5.14.0.jar:5.14.0]
        at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)[activemq-client-5.14.0.jar:5.14.0]
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:333)[activemq-broker-5.14.0.jar:5.14.0]
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:197)[activemq-broker-5.14.0.jar:5.14.0]
        at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.14.0.jar:5.14.0]
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)[activemq-client-5.14.0.jar:5.14.0]
        at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)[activemq-client-5.14.0.jar:5.14.0]
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.14.0.jar:5.14.0]
        at org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:108)[activemq-client-5.14.0.jar:5.14.0]
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.14.0.jar:5.14.0]
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.14.0.jar:5.14.0]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_101]
{noformat}

Empty journal files:

{noformat}
-rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51425.log
-rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51426.log
-rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51427.log
-rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51428.log
-rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51429.log
{noformat}

lsof output:

{noformat}
java    29263 activemq  452u      REG             202,48         0       160 /activemq/kahadb/db-51425.log
java    29263 activemq  455u      REG             202,48         0       153 /activemq/kahadb/db-51426.log
java    29263 activemq  456u      REG             202,48         0       133 /activemq/kahadb/db-51427.log
java    29263 activemq  462u      REG             202,48         0       157 /activemq/kahadb/db-51428.log
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message