Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
From: =?utf-8?Q?Michael_Andr=C3=A9_Pearce?= <michael.andre.pearce@me.com>
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: quoted-printable
MIME-version: 1.0 (1.0)
Date: Fri, 02 Jun 2017 17:57:23 +0100
Subject: Re: artemis 2.2.0 logging disaster
Message-id: <FD59B700-43BE-47F4-89F1-206A8EBE6FFF@me.com>
References: <1496393053.6455.18.camel@waastad.org>
In-reply-to: <1496393053.6455.18.camel@waastad.org>
To: users@activemq.apache.org
archived-at: Fri, 02 Jun 2017 16:57:39 -0000

Essentially just from this log output I assume the server your broker is run=
ning out of ram to use.=20
This can be either=20
A) genuine memory leak in artemis
B) you simply don't have enough ram for the load/throughout.

Some questions:

Is the load constant?=20
Do you have server ram usage metrics available?

You should ensure there is more ram available to the broker instance than ju=
st heap allocation, for network buffers etc.

Cheers
Mike


Sent from my iPhone

> On 2 Jun 2017, at 09:44, Helge Waastad <helge@waastad.org> wrote:
>=20
> Hi,
> I'm running artemis 2.2.0 as a docker container.
>=20
> I'm collecting MQTT messages an these are consumed by a JMS consumer
> (artemis-jms-client)
>=20
> It's running fine for a while, but suddenly this appear (docker *-
> json.log):
>=20
> {"log":"19:16:12,338 WARN=20
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection]=20
> Trying to allocate 712 bytes, System is throwing OutOfMemoryError on=20
> NettyConnection org.apache.activemq.art
> emis.core.remoting.impl.netty.NettyServerConnection@6f035b0a[local=3D=20
> /10.42.154.105:61616, remote=3D/10.42.21.198:40844], there are currently=20=

> pendingWrites: [NETTY] -\u003e 0[EVENT LOOP] -\u003e 0 causes: fail
> ed to allocate 16777216 byte(s) of direct memory (used: 1057466368,
> max:=20
> 1073741824): io.netty.util.internal.OutOfDirectMemoryError: failed to=20
> allocate 16777216 byte(s) of direct memory (used: 1057466368, m
> ax:=20
> 1073741824)\r\n","stream":"stdout","time":"2017-06-
> 01T19:16:12.342853929Z"}
> {"log":"19:16:12,342 WARN  [org.apache.activemq.artemis.core.server]=20
> AMQ222151: removing consumer which did not handle a message,=20
> consumer=3DServerConsumerImpl [id=3D0, filter=3Dnull,
> binding=3DLocalQueueBinding [a
> ddress=3DCentreonTopic, queue=3DQueueImpl[name=3DCentreonTopic,=20
> postOffice=3DPostOfficeImpl=20
> [server=3DActiveMQServerImpl::serverUUID=3D772ad6f8-4630-11e7-93cd-
> 02a837635b7b],=20
> temp=3Dfalse]@3e389a2d, filter=3Dnull, name=3DCent
> reonTopic,=20
> clusterName=3DCentreonTopic772ad6f8-4630-11e7-93cd-02a837635b7b]],=20
> message=3DReference[715739]:NON-
> RELIABLE:CoreMessage[messageID=3D715739,durable=3Dfalse,userID=3Dnull,prio=
rit
> y=3D0,=20
> timestamp=3D0,expiration=3D0
> , durable=3Dfalse,=20
> address=3DCentreonTopic,properties=3DTypedProperties[mqtt.message.retain=3D=
fa
> lse,mqtt.qos.level=3D0]]@1623021181:=20
> io.netty.util.internal.OutOfDirectMemoryError: failed to allocate=20
> 16777216 byte(s)
> of direct memory (used: 1057466368, max:=20
> 1073741824)\r\n","stream":"stdout","time":"2017-06-
> 01T19:16:12.347107296Z"}
> {"log":"19:31:54,236 WARN=20
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection]=20
> Trying to allocate 548 bytes, System is throwing OutOfMemoryError on=20
> NettyConnection org.apache.activemq.art
> emis.core.remoting.impl.netty.NettyServerConnection@7b18e1a6[local=3D=20
> /10.42.154.105:61616, remote=3D/10.42.162.183:48376], there are
> currently=20
> pendingWrites: [NETTY] -\u003e 0[EVENT LOOP] -\u003e 0 causes: fai
> led to allocate 16777216 byte(s) of direct memory (used: 1057466368,=20
> max: 1073741824): io.netty.util.internal.OutOfDirectMemoryError:
> failed=20
> to allocate 16777216 byte(s) of direct memory (used: 1057466368,
> max:=20
> 1073741824)\r\n","stream":"stdout","time":"2017-06-
> 01T19:31:54.238904544Z"}
> {"log":"19:31:54,238 WARN  [org.apache.activemq.artemis.core.server]=20
> AMQ222151: removing consumer which did not handle a message,=20
> consumer=3DServerConsumerImpl [id=3D0, filter=3Dnull,
> binding=3DLocalQueueBinding [a
> ddress=3DCentreonTopic, queue=3DQueueImpl[name=3DCentreonTopic,=20
> postOffice=3DPostOfficeImpl=20
> [server=3DActiveMQServerImpl::serverUUID=3D772ad6f8-4630-11e7-93cd-
> 02a837635b7b],=20
> temp=3Dfalse]@3e389a2d, filter=3Dnull, name=3DCent
> reonTopic,=20
> clusterName=3DCentreonTopic772ad6f8-4630-11e7-93cd-02a837635b7b]],=20
> message=3DReference[722892]:NON-
> RELIABLE:CoreMessage[messageID=3D722892,durable=3Dfalse,userID=3Dnull,prio=
rit
> y=3D0,=20
> timestamp=3D0,expiration=3D0
> , durable=3Dfalse,=20
> address=3DCentreonTopic,properties=3DTypedProperties[mqtt.message.retain=3D=
fa
> lse,mqtt.qos.level=3D0]]@1252621657:=20
> io.netty.util.internal.OutOfDirectMemoryError: failed to allocate=20
> 16777216 byte(s)
> of direct memory (used: 1057466368, max:=20
> 1073741824)\r\n","stream":"stdout","time":"2017-06-
> 01T19:31:54.239955162Z"}
>=20
>=20
>=20
> Then after a couple of hours:
>=20
> {"log":"23:22:24,013 WARN  [io.netty.channel.DefaultChannelPipeline]
> An=20
> exceptionCaught() event was fired, and it reached at the tail of the=20
> pipeline. It usually means the last handler in the pipeline did n
> ot handle the exception.:
> io.netty.util.internal.OutOfDirectMemoryError:=20
> failed to allocate 16777216 byte(s) of direct memory (used:
> 1057466368,=20
> max: 1073741824)\r\n","stream":"stdout","time":"2017-06-01T23
> :22:24.015087347Z"}
> {"log":"23:22:24,014 WARN  [io.netty.channel.DefaultChannelPipeline]
> An=20
> exceptionCaught() event was fired, and it reached at the tail of the=20
> pipeline. It usually means the last handler in the pipeline did n
> ot handle the exception.:
> io.netty.util.internal.OutOfDirectMemoryError:=20
> failed to allocate 16777216 byte(s) of direct memory (used:
> 1057466368,=20
> max: 1073741824)\r\n","stream":"stdout","time":"2017-06-01T23
> :22:24.015759902Z"}
> {"log":"23:22:24,015 WARN  [io.netty.channel.DefaultChannelPipeline]
> An=20
> exceptionCaught() event was fired, and it reached at the tail of the=20
> pipeline. It usually means the last handler in the pipeline did n
> ot handle the exception.:
> io.netty.util.internal.OutOfDirectMemoryError:=20
> failed to allocate 16777216 byte(s) of direct memory (used:
> 1057466368,=20
> max: 1073741824)\r\n","stream":"stdout","time":"2017-06-01T23
> :22:24.016623101Z"}
>=20
>=20
> And this message is looping and in 5 mins it's filled my 12GB drive.
>=20
> Any clues what to do? I'll do some more debugging.
>=20
> /hw