activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-6451) Preallocation strategy of zeros can leak file descriptors causing crash
Date Tue, 08 Nov 2016 14:06:58 GMT

    [ https://issues.apache.org/jira/browse/AMQ-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15647652#comment-15647652
] 

ASF subversion and git services commented on AMQ-6451:
------------------------------------------------------

Commit 3b7613d9300443898b608a52cb4e763821fdc163 in activemq's branch refs/heads/master from
[~gtully]
[ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=3b7613d ]

AMQ-6451 - catch errors on preallocation. allocate direct buffer on start and reuse to ensure
resource availability at runtime


> Preallocation strategy of zeros can leak file descriptors causing crash
> -----------------------------------------------------------------------
>
>                 Key: AMQ-6451
>                 URL: https://issues.apache.org/jira/browse/AMQ-6451
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: KahaDB
>    Affects Versions: 5.14.0
>            Reporter: Cott Lang
>            Assignee: Gary Tully
>            Priority: Critical
>
> If the preallocationStrategy is set to 'zeros', ActiveMQ can intermittently become unable
to allocate direct buffer memory with the default JVM settings. The exception isn't handled,
and ends up both creating an empty journal file and, more importantly, leaking a file descriptor.
> ActiveMQ eventually runs out of file descriptors and crashes.
> In addition to handling this condition, perhaps the default ACTIVEMQ_OPTS_MEMORY settings
should configure enough direct memory to allow some multiple of log files to be created near
simultaneously, or at least this possibility documented in the KahaDB settings.
> Relevant logs:
> {noformat}
> 2016-10-03 12:47:24,154 | WARN  | Async error occurred: java.lang.OutOfMemoryError: Direct
buffer memory | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
ssl:///x.x.x.x:60805
> 2016-10-03 12:47:24,818 | WARN  | Async error occurred: java.lang.OutOfMemoryError: Direct
buffer memory | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
ssl:///x.x.x.x:60811
> 2016-10-03 12:47:25,477 | WARN  | Async error occurred: java.lang.OutOfMemoryError: Direct
buffer memory | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
ssl:///x.x.x.x:49830
> 2016-10-03 12:47:26,146 | WARN  | Store COMMIT FAILED:  | org.apache.activemq.transaction.LocalTransaction
| ActiveMQ Transport: ssl:///x.x.x.x:65534
> java.lang.OutOfMemoryError: Direct buffer memory
>         at java.nio.Bits.reserveMemory(Bits.java:693)[:1.8.0_101]
>         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)[:1.8.0_101]
>         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)[:1.8.0_101]
>         at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)[:1.8.0_101]
>         at sun.nio.ch.IOUtil.write(IOUtil.java:58)[:1.8.0_101]
>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)[:1.8.0_101]
>         at org.apache.activemq.store.kahadb.disk.journal.Journal.doPreallocationZeros(Journal.java:366)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.disk.journal.Journal.preallocateEntireJournalDataFile(Journal.java:333)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.disk.journal.Journal.newDataFile(Journal.java:631)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.disk.journal.Journal.rotateWriteFile(Journal.java:595)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.disk.journal.Journal.getCurrentDataFile(Journal.java:984)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.enqueue(DataFileAppender.java:189)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:128)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:890)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1108)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1090)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.store.kahadb.KahaDBTransactionStore.commit(KahaDBTransactionStore.java:301)[activemq-kahadb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:70)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:252)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:113)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:113)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:118)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:535)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)[activemq-client-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:333)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:197)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.14.0.jar:5.14.0]
>         at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)[activemq-client-5.14.0.jar:5.14.0]
>         at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)[activemq-client-5.14.0.jar:5.14.0]
>         at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.14.0.jar:5.14.0]
>         at org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:108)[activemq-client-5.14.0.jar:5.14.0]
>         at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.14.0.jar:5.14.0]
>         at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.14.0.jar:5.14.0]
>         at java.lang.Thread.run(Thread.java:745)[:1.8.0_101]
> {noformat}
> Empty journal files:
> {noformat}
> -rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51425.log
> -rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51426.log
> -rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51427.log
> -rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51428.log
> -rw-r--r-- 1 activemq activemq         0 Oct  3 12:47 db-51429.log
> {noformat}
> lsof output:
> {noformat}
> java    29263 activemq  452u      REG             202,48         0       160 /activemq/kahadb/db-51425.log
> java    29263 activemq  455u      REG             202,48         0       153 /activemq/kahadb/db-51426.log
> java    29263 activemq  456u      REG             202,48         0       133 /activemq/kahadb/db-51427.log
> java    29263 activemq  462u      REG             202,48         0       157 /activemq/kahadb/db-51428.log
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message