activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilkka Virolainen <Ilkka.Virolai...@bitwise.fi>
Subject RE: Artemis 2.4.0 - Issues with memory leaks and JMS message redistribution
Date Thu, 22 Feb 2018 11:38:09 GMT
To answer my own question in case anyone else is wondering about a similar issue, turns out
the change in addressing is referred in ticket [1] and adding the multicastPrefix and anycastPrefix
described in the ticket to my broker acceptors seems to have fixed my problem. If the issue
regarding memory leaks persists I will try to provide a reproducible test case.

Thank you for your help, Justin.

Best regards,
- Ilkka

[1] https://issues.apache.org/jira/browse/ARTEMIS-1644


-----Original Message-----
From: Ilkka Virolainen [mailto:Ilkka.Virolainen@bitwise.fi] 
Sent: 22. helmikuuta 2018 12:33
To: users@activemq.apache.org
Subject: RE: Artemis 2.4.0 - Issues with memory leaks and JMS message redistribution

Having removed the address configuration and having switched from 2.4.0 to yesterday's snapshot
of 2.5.0 it seems like the redistribution of messages is now working, but there also seems
to have been a change in addressing between the versions causing another problem related to
jms.queue / jms.topic prefixing. While the NMS clients listen and artemis jms clients send
to the same topics as described in the previous message, Artemis 2.5.0 prefixes the addresses
with jms.topic. While the messages are being sent to e.g.  A.B.f64dd592-a8fb-442e-826d-927834d566f4.C.D
they are only received if I explicitly prefix the listening address with jms.topic, for example
topic://jms.topic.A.B.*.C.D. Can this somehow be avoided in the broker configuration?

Best regards

-----Original Message-----
From: Justin Bertram [mailto:jbertram@apache.org]
Sent: 21. helmikuuta 2018 15:19
To: users@activemq.apache.org
Subject: Re: Artemis 2.4.0 - Issues with memory leaks and JMS message redistribution

Your first issue is probably a misconfiguration.  Your cluster-connection is using an "address"
value of '*' which I assume is supposed to mean "all addresses," but the "address" element
doesn't support wildcards like this.
Just leave it empty to match all addresses.  See the documentation [1] for more details.

Even after you fix that configuration issue you may run into issues.  These may be fixed already
via ARTEMIS-1523 and/or ARTEMIS-1680.  If you have a reproducible test-case then you can verify
using the head of the master branch.

For the memory issue it would be helpful to have some heap dumps or something to actually
see what's actually consuming the memory.  Better yet would be a reproducible test-case. 
Do you have either?


Justin

[1] https://activemq.apache.org/artemis/docs/latest/clusters.html



On Wed, Feb 21, 2018 at 5:39 AM, Ilkka Virolainen < Ilkka.Virolainen@bitwise.fi> wrote:

> Hello,
>
> I am using Artemis 2.4.0 to broker messages through JMS queues/topics 
> between a set of clients. Some are Apache NMS 1.7.2 ActiveMQ clients 
> and others are using Artemis JMS client 1.5.4 included in Spring Boot 1.5.3.
> Broker topology is a symmetric cluster of two live nodes with static 
> connectors, both nodes having been setup as replicating colocated 
> backup pairs with scale down. I have two quite frustrating issues at the moment:
> message redistribution not working correctly and a memory leak causing 
> eventual thread death.
>
> ISSUE #1 - Message redistribution / load balancing not working:
>
> Client 1 (NMS) connects to broker a and starts listening, artemis 
> creates the following address:
>
> (Broker a):
> A.B.*.C.D
> |-queues
> |-multicast
>   |-f64dd592-a8fb-442e-826d-927834d566f4
>
> Server 1 (artemis-jms-client) connects to broker b and sends a message 
> to
> topic: A.B.f64dd592-a8fb-442e-826d-927834d566f4.C.D - this should be 
> routed to broker a since the corresponding queue has no consumers on 
> broker b (the queue does not exist). This however does not happen and 
> the client receives no messages. Broker b has some other clients 
> connected, causing similar (but not the same) queues having been created:
>
> (Broker b):
> A.B.*.C.D
> |-queues
> |-multicast
>   |-1eb48079-7fd8-40e9-b822-bcc25695ced0
>   |-9f295257-c352-4ae6-b74b-d5994f330485
>
>
> ISSUE #2: - Memory leak and eventual thread death
>
> Artemis broker has 4GB allocated heap space and global-max-size is set 
> up as half of that (being the default setting). Address-full-policy is 
> set to PAGE for all addresses and some individual addresses have small 
> max-size-bytes values set e.g. 104857600. As far as I know the paging 
> settings should limit memory usage but what happens is that at times 
> Artemis uses the whole heap space, encounters an out of memory error 
> and
> dies:
>
> 05:39:29,510 WARN  [org.eclipse.jetty.util.thread.QueuedThreadPool] :
> java.lang.OutOfMemoryError: Java heap space
> 05:39:16,646 WARN  [io.netty.channel.ChannelInitializer] Failed to 
> initialize a channel. Closing: [id: ...]: java.lang.OutOfMemoryError:
> Java heap space
> 05:41:05,597 WARN  [org.eclipse.jetty.util.thread.QueuedThreadPool]
> Unexpected thread death: org.eclipse.jetty.util.thread.
> QueuedThreadPool$2@5ffaba31 in qtp20111564{STARTED,8<=8<=200,i=2,q=0}
>
> Are these known issues in Artemis or misconfigurations in the brokers?
>
> The broker configurations are as follows. Broker b has an identical 
> configuration excluding that the cluster connector's connector-ref and 
> static-connector connector-ref refer to broker b and broker a respectively.
>
> Best regards,
>
> broker.xml (broker a):
>
> <?xml version='1.0'?>
> <configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/ 
> 2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq 
> /schema/artemis-configuration.xsd">
>     <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/ 
> 2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core ">
>         <name>[broker-a-ip]</name>
>         <persistence-enabled>true</persistence-enabled>
>
>         <journal-type>NIO</journal-type>
>
>         <paging-directory>...</paging-directory>
>         <bindings-directory>...</bindings-directory>
>         <journal-directory>...</journal-directory>
>         <large-messages-directory>...</large-messages-directory>
>
>         <journal-datasync>true</journal-datasync>
>         <journal-min-files>2</journal-min-files>
>         <journal-pool-files>-1</journal-pool-files>
>         <journal-buffer-timeout>788000</journal-buffer-timeout>
>         <disk-scan-period>5000</disk-scan-period>
>
>         <max-disk-usage>97</max-disk-usage>
>
>         <critical-analyzer>true</critical-analyzer>
>         <critical-analyzer-timeout>120000</critical-analyzer-timeout>
>         <critical-analyzer-check-period>60000</critical-
> analyzer-check-period>
>         <critical-analyzer-policy>HALT</critical-analyzer-policy>
>
>         <acceptors>
>             <acceptor name="invm-acceptor">vm://0</acceptor>
>             <acceptor name="artemis">tcp://0.0.0.0:61616</acceptor>
>             <acceptor name="ssl">tcp://0.0.0.0:61617?sslEnabled=true;
> keyStorePath=...;keyStorePassword=...</acceptor>
>         </acceptors>
>         <connectors>
>             <connector name="invm-connector">vm://0</connector>
>             <connector name="netty-connector">tcp://[ 
> broker-a-ip]:61616</connector>
>             <connector name="broker-b-connector">[ 
> broker-b-ip]:61616</connector>
>         </connectors>
>
>         <cluster-connections>
>             <cluster-connection name="cluster-name">
>                 <address>*</address>
>                 <connector-ref>netty-connector</connector-ref>
>                 <retry-interval>500</retry-interval>
>                 <reconnect-attempts>5</reconnect-attempts>
>                 <use-duplicate-detection>true</use-duplicate-detection>
>                 <message-load-balancing>ON_DEMAND</message-load-balancing>
>                 <max-hops>1</max-hops>
>                 <static-connectors>
>                     <connector-ref>broker-b-connector</connector-ref>
>                 </static-connectors>
>             </cluster-connection>
>         </cluster-connections>
>
>         <ha-policy>
>             <replication>
>                 <colocated>
>                     
> <backup-request-retry-interval>5000</backup-request-
> retry-interval>
>                     <max-backups>3</max-backups>
>                     <request-backup>true</request-backup>
>                     <backup-port-offset>100</backup-port-offset>
>                     <excludes>
>                         <connector-ref>invm-connector</connector-ref>
>                         <connector-ref>netty-connector</connector-ref>
>                     </excludes>
>                     <master>
>                         <check-for-live-server>true</
> check-for-live-server>
>                     </master>
>                     <slave>
>                         <restart-backup>false</restart-backup>
>                         <scale-down />
>                     </slave>
>                 </colocated>
>             </replication>
>         </ha-policy>
>
>         <cluster-user>ARTEMIS.CLUSTER.ADMIN.USER</cluster-user>
>         <cluster-password>[the shared cluster 
> password]</cluster-password>
>
>         <security-settings>
>             <security-setting match="#">
>                 <permission type="createDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="deleteDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="createNonDurableQueue" roles="amq, 
> other-role"  />
>                 <permission type="createAddress" roles="amq, other-role" />
>                 <permission type="deleteNonDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="deleteAddress" roles="amq, other-role" />
>                 <permission type="consume" roles="amq, other-role" />
>                 <permission type="browse" roles="amq, other-role" />
>                 <permission type="send" roles="amq, other-role" />
>                 <permission type="manage" roles="amq" />
>             </security-setting>
>             <security-setting match="A.some.queue">
>                 <permission type="createNonDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="deleteNonDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="createDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="deleteDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="createAddress" roles="amq, other-role" />
>                 <permission type="deleteAddress" roles="amq, other-role" />
>                 <permission type="consume" roles="amq, other-role" />
>                 <permission type="browse" roles="amq, other-role" />
>                 <permission type="send" roles="amq, other-role" />
>             </security-setting>
>                 <security-setting match="A.some.other.queue">
>                 <permission type="createNonDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="deleteNonDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="createDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="deleteDurableQueue" roles="amq, 
> other-role" />
>                 <permission type="createAddress" roles="amq, other-role" />
>                 <permission type="deleteAddress" roles="amq, other-role" />
>                 <permission type="consume" roles="amq, other-role" />
>                 <permission type="browse" roles="amq, other-role" />
>                 <permission type="send" roles="amq, other-role" />
>             </security-setting>
>             ...
>             ... etc.
>             ...
>         </security-settings>
>
>         <address-settings>
>             <address-setting match="activemq.management#">
>                 <dead-letter-address>DLQ</dead-letter-address>
>                 <expiry-address>ExpiryQueue</expiry-address>
>                 <redelivery-delay>0</redelivery-delay>
>                 <max-size-bytes>-1</max-size-bytes>
>                 
> <message-counter-history-day-limit>10</message-counter-
> history-day-limit>
>                 <address-full-policy>PAGE</address-full-policy>
>             </address-setting>
>             <!--default for catch all -->
>             <address-setting match="#">
>                 <dead-letter-address>DLQ</dead-letter-address>
>                 <expiry-address>ExpiryQueue</expiry-address>
>                 <redelivery-delay>0</redelivery-delay>
>                 <max-size-bytes>-1</max-size-bytes>
>                 
> <message-counter-history-day-limit>10</message-counter-
> history-day-limit>
>                 <address-full-policy>PAGE</address-full-policy>
>                 <redistribution-delay>1000</redistribution-delay>
>             </address-setting>
>             <address-setting match="DLQ">
>                 <!-- 100 * 1024 * 1024 -> 100MB -->
>                 <max-size-bytes>104857600</max-size-bytes>
>                 <!-- 1000 * 60 * 60 -> 1h -->
>                 <expiry-delay>3600000</expiry-delay>
>                 <expiry-address />
>             </address-setting>
>             <address-setting match="A.some.queue">
>                 <redelivery-delay-multiplier>1.0</redelivery-delay-
> multiplier>
>                 <redelivery-delay>0</redelivery-delay>
>                 <max-redelivery-delay>10</max-redelivery-delay>
>             </address-setting>
>                 <address-setting match="A.some.other.queue">
>                 <redelivery-delay-multiplier>1.0</redelivery-delay-
> multiplier>
>                 <redelivery-delay>0</redelivery-delay>
>                 <max-redelivery-delay>10</max-redelivery-delay>
>                 <max-delivery-attempts>1</max-delivery-attempts>
>                 <max-size-bytes>104857600</max-size-bytes>
>             </address-setting>
>             ...
>             ... etc.
>             ...
>         </address-settings>
>
>         <addresses>
>             <address name="DLQ">
>                 <anycast>
>                     <queue name="DLQ" />
>                 </anycast>
>             </address>
>             <address name="ExpiryQueue">
>                 <anycast>
>                     <queue name="ExpiryQueue" />
>                 </anycast>
>             </address>
>             <address name="A.some.queue">
>                 <anycast>
>                     <queue name="A.some.queue">
>                         <durable>true</durable>
>                     </queue>
>                 </anycast>
>             </address>
>             <address name="A.some.other.queue">
>                 <anycast>
>                     <queue name="A.some.other.queue">
>                         <durable>true</durable>
>                     </queue>
>                 </anycast>
>             </address>
>             ...
>             ... etc.
>             ...
>         </addresses>
>     </core>
> </configuration>
>
Mime
View raw message