Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 173D1200CC8 for ; Fri, 14 Jul 2017 21:39:13 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1598716E50D; Fri, 14 Jul 2017 19:39:13 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0CC2516E50A for ; Fri, 14 Jul 2017 21:39:11 +0200 (CEST) Received: (qmail 79982 invoked by uid 500); 14 Jul 2017 19:39:11 -0000 Mailing-List: contact issues-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list issues@activemq.apache.org Received: (qmail 79973 invoked by uid 99); 14 Jul 2017 19:39:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jul 2017 19:39:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id B913B1A0745 for ; Fri, 14 Jul 2017 19:39:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id ng-QX_usEzQV for ; Fri, 14 Jul 2017 19:39:07 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 1A2F35FCD7 for ; Fri, 14 Jul 2017 19:39:07 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 75854E0D60 for ; Fri, 14 Jul 2017 19:39:05 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id BABCF24765 for ; Fri, 14 Jul 2017 19:39:02 +0000 (UTC) Date: Fri, 14 Jul 2017 19:39:02 +0000 (UTC) From: "clebert suconic (JIRA)" To: issues@activemq.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 14 Jul 2017 19:39:13 -0000 [ https://issues.apache.org/jira/browse/ARTEMIS-1286?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16= 087899#comment-16087899 ]=20 clebert suconic commented on ARTEMIS-1286: ------------------------------------------ did you run against master? there was a recent fix recently, and I'm about = to release. other than that.. if you still see an isssue, please attach your test.. the= n we will know if it's a bug or something else. > Server stops responding and throws OutOfDirectMemoryError when sending & = receiving lots of 2MB messages. > -------------------------------------------------------------------------= ------------------------------- > > Key: ARTEMIS-1286 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1286 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker, MQTT > Affects Versions: 2.1.0, 2.2.0 > Reporter: Phillip Jenkins > Priority: Blocker > > Originally seen in v2.1 and present in v2.2. > If you send and receive a lot of 2MB messages in short time thru Artemis = via Netty connector, the server throws the following OutOfDirectMemoryError= exception. The server stops responding and will not (any longer) accept co= nnections without throwing exceptions in an infinite loop. > {code:java} > 11:24:07,434 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server s= tarted at http://localhost:8161 > 11:24:07,434 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolok= ia REST API available at http://localhost:8161/jolokia > 11:58:35,991 WARN [org.apache.activemq.artemis.core.server] AMQ222151: r= emoving consumer which did not handle a message, consumer=3DServerConsumerI= mpl [id=3D2229, filter=3Dnull, binding=3DLocalQueueBinding [address=3DSOAP.= S.PRN.ACBCAF6238234680, queue=3DQueueImpl[name=3DACBCAF6238234680.SOAP.S.PR= N.ACBCAF6238234680, postOffice=3DPostOfficeImpl [server=3DActiveMQServerImp= l::serverUUID=3Dafa7de73-67e7-11e7-a231-54ee7505882d], temp=3Dfalse]@46adc2= a5, filter=3DFilterImpl [sfilterString=3DNOT ((AMQAddress =3D 'activemq.man= agement') OR (AMQAddress =3D 'activemq.notifications'))], name=3DACBCAF6238= 234680.SOAP.S.PRN.ACBCAF6238234680, clusterName=3DACBCAF6238234680.SOAP.S.P= RN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]], message=3DRefere= nce[3551]:NON-RELIABLE:CoreMessage[messageID=3D3551,durable=3Dfalse,userID= =3Dnull,priority=3D0, timestamp=3D0,expiration=3D0, durable=3Dfalse, addres= s=3DSOAP.S.PRN.ACBCAF6238234680,properties=3DTypedProperties[mqtt.message.r= etain=3Dfalse,mqtt.qos.level=3D0]]@1305748199: io.netty.util.internal.OutOf= DirectMemoryError: failed to allocate 3729415 byte(s) of direct memory (use= d: 1070952692, max: 1073741824) > =09at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(Pla= tformDependent.java:585) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(Pl= atformDependent.java:539) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect= (UnpooledUnsafeNoCleanerDirectByteBuf.java:30) [netty-all-4.1.9.Final.jar:4= .1.9.Final] > =09at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafe= NoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169) [n= etty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDi= rectByteBuf.java:68) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(Unpoole= dUnsafeNoCleanerDirectByteBuf.java:25) [netty-all-4.1.9.Final.jar:4.1.9.Fin= al] > =09at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafe= NoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164) [netty-all= -4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledBy= teBufAllocator.java:73) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteB= ufAllocator.java:181) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllo= cator.java:117) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828)= [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) [= netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.= readBytes(ChannelBufferWrapper.java:315) [artemis-commons-2.2.0-SNAPSHOT.ja= r:2.2.0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.s= endServerMessage(MQTTPublishManager.java:277) [artemis-mqtt-protocol-2.2.0-= SNAPSHOT.jar:] > =09at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.s= endMessage(MQTTPublishManager.java:121) [artemis-mqtt-protocol-2.2.0-SNAPSH= OT.jar:] > =09at org.apache.activemq.artemis.core.protocol.mqtt.MQTTSessionCallback.= sendMessage(MQTTSessionCallback.java:52) [artemis-mqtt-protocol-2.2.0-SNAPS= HOT.jar:] > =09at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.del= iverStandardMessage(ServerConsumerImpl.java:1078) [artemis-server-2.2.0-SNA= PSHOT.jar:2.2.0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.pro= ceedDeliver(ServerConsumerImpl.java:450) [artemis-server-2.2.0-SNAPSHOT.jar= :2.2.0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliv= er(QueueImpl.java:2688) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliverDirec= t(QueueImpl.java:2673) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.server.impl.QueueImpl.addTail(Queu= eImpl.java:645) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.add= References(PostOfficeImpl.java:1297) [artemis-server-2.2.0-SNAPSHOT.jar:2.2= .0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.acc= ess$000(PostOfficeImpl.java:93) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SN= APSHOT] > =09at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl$1.d= one(PostOfficeImpl.java:1157) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAP= SHOT] > =09at org.apache.activemq.artemis.core.persistence.impl.journal.Operation= ContextImpl.executeOnCompletion(OperationContextImpl.java:181) [artemis-ser= ver-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.persistence.impl.journal.Operation= ContextImpl.executeOnCompletion(OperationContextImpl.java:130) [artemis-ser= ver-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJ= ournalStorageManager.afterCompleteOperations(AbstractJournalStorageManager.= java:303) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.pro= cessRoute(PostOfficeImpl.java:1149) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.= 0-SNAPSHOT] > =09at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.rou= te(PostOfficeImpl.java:787) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSH= OT] > =09at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.rou= te(PostOfficeImpl.java:689) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSH= OT] > =09at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSe= nd(ServerSessionImpl.java:1674) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SN= APSHOT] > =09at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send= (ServerSessionImpl.java:1351) [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAP= SHOT] > =09at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.s= endInternal(MQTTPublishManager.java:166) [artemis-mqtt-protocol-2.2.0-SNAPS= HOT.jar:] > =09at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.h= andleMessage(MQTTPublishManager.java:136) [artemis-mqtt-protocol-2.2.0-SNAP= SHOT.jar:] > =09at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.= handlePublish(MQTTProtocolHandler.java:204) [artemis-mqtt-protocol-2.2.0-SN= APSHOT.jar:] > =09at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.= channelRead(MQTTProtocolHandler.java:118) [artemis-mqtt-protocol-2.2.0-SNAP= SHOT.jar:] > =09at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Ab= stractChannelHandlerContext.java:362) [netty-all-4.1.9.Final.jar:4.1.9.Fina= l] > =09at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Ab= stractChannelHandlerContext.java:348) [netty-all-4.1.9.Final.jar:4.1.9.Fina= l] > =09at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abst= ractChannelHandlerContext.java:340) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToM= essageDecoder.java:293) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessa= geDecoder.java:267) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Ab= stractChannelHandlerContext.java:362) [netty-all-4.1.9.Final.jar:4.1.9.Fina= l] > =09at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Ab= stractChannelHandlerContext.java:348) [netty-all-4.1.9.Final.jar:4.1.9.Fina= l] > =09at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abst= ractChannelHandlerContext.java:340) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(Def= aultChannelPipeline.java:1334) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Ab= stractChannelHandlerContext.java:362) [netty-all-4.1.9.Final.jar:4.1.9.Fina= l] > =09at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Ab= stractChannelHandlerContext.java:348) [netty-all-4.1.9.Final.jar:4.1.9.Fina= l] > =09at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChan= nelPipeline.java:926) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(Abst= ractNioByteChannel.java:134) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.j= ava:624) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioE= ventLoop.java:559) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.= java:476) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) [netty= -all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThre= adEventExecutor.java:858) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72] > {code} > Subsequent attempts to connect with the server will fail and causes an in= finite loop throwing the following exception into the log until the server = is shut down. > {code:java} > 12:10:31,004 WARN [io.netty.channel.DefaultChannelPipeline] An exception= Caught() event was fired, and it reached at the tail of the pipeline. It us= ually means the last handler in the pipeline did not handle the exception.:= io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216= byte(s) of direct memory (used: 1070952692, max: 1073741824) > =09at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(Pla= tformDependent.java:585) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(Pl= atformDependent.java:539) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java= :760) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:736) = [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) [netty= -all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) [netty-all-4= .1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) [netty-all-4= .1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBu= fAllocator.java:320) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteB= ufAllocator.java:181) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteB= ufAllocator.java:172) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAl= locator.java:133) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageH= andle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:80) [netty-all-4= .1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(Abst= ractNioByteChannel.java:122) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.j= ava:624) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioE= ventLoop.java:559) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.= java:476) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) [netty= -all-4.1.9.Final.jar:4.1.9.Final] > =09at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThre= adEventExecutor.java:858) [netty-all-4.1.9.Final.jar:4.1.9.Final] > =09at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72] > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)