From users-return-51493-archive-asf-public=cust-asf.ponee.io@activemq.apache.org Tue Jun 11 16:21:52 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 43280180627 for ; Tue, 11 Jun 2019 18:21:52 +0200 (CEST) Received: (qmail 33466 invoked by uid 500); 11 Jun 2019 16:21:51 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 33455 invoked by uid 99); 11 Jun 2019 16:21:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jun 2019 16:21:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 195E11A4323 for ; Tue, 11 Jun 2019 16:21:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.738 X-Spam-Level: **** X-Spam-Status: No, score=4.738 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, FORGED_GMAIL_RCVD=1, FREEMAIL_ENVFROM_END_DIGIT=0.25, KAM_SHORT=0.001, NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_SOFTFAIL=0.972, URI_HEX=1.313, URI_TRY_3LD=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id gONmx7ipKapj for ; Tue, 11 Jun 2019 16:21:46 +0000 (UTC) Received: from n4.nabble.com (n4.nabble.com [199.38.86.66]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id C12E05F438 for ; Tue, 11 Jun 2019 16:21:45 +0000 (UTC) Received: from n4.nabble.com (localhost [127.0.0.1]) by n4.nabble.com (Postfix) with ESMTP id 14DB18EDA28D for ; Tue, 11 Jun 2019 11:21:45 -0500 (CDT) Date: Tue, 11 Jun 2019 11:21:44 -0500 (CDT) From: hannibal To: users@activemq.apache.org Message-ID: <1560270104741-0.post@n4.nabble.com> In-Reply-To: References: <1559750306854-0.post@n4.nabble.com> <1559899104801-0.post@n4.nabble.com> Subject: Re: Artemis 2.7.0 - Shared Store cluster - RHEL guest/WIN host VMs - Failover not working MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi, some changes. The first issue that I figured out on my broker.xml files was the leading "/" on all file locations. This was preventing the slave to share the same storage as the master. Then I moved to 2.9.0. But I'm still experiencing weird behaviour. The client application connects randomly to the master or to the client. Usually it ends with consumers on one and producers on the other so that nothing goes as it should (what is being produced is also consumed). Here how the client application connects: public static final String ARTEMIS_CLUSTER_URL_1M = "(tcp://artemis-1M:61616,tcp://artemis-1S:61616)?ha=true&retryInterval=1000&reconnectAttempts=-1"; public static final String ARTEMIS_CLUSER_USERNAME = "ACTIVEMQ.CLUSTER.ADMIN.USER"; public static final String ARTEMIS_CLUSTER_PASSWORD = "XxXxX"; ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(urlConfiguration, runCfg.ARTEMIS_CLUSER_USERNAME,runCfg.ARTEMIS_CLUSTER_PASSWORD ); ... I'm sure I'm missing something but I cannot figure out what. Also, master/slave behaviour looks to me very different from what I was used to with ActiveMQ. Again the artemis-1M (master) broker configuration: artemis-1M true ASYNCIO /vagrant_data/data/paging /vagrant_data/data/bindings /vagrant_data/data/journal /vagrant_data/data/large-messages true 2 10 10M 16000 4096 5000 90 true 120000 60000 HALT true ACTIVEMQ.CLUSTER.ADMIN.USER XxXxX artemis-1M true artemis-1S tcp://artemis-1M:61616 tcp://artemis-1S:61616 tcp://0.0.0.0:61616 DLQ ExpiryQueue 0 -1 10 PAGE true true true true DLQ ExpiryQueue 0 -1 10 PAGE true true true true 1000
-------------------------------- The artemis-1S (slave) broker configuration: artemis-1M true ASYNCIO /vagrant_data/data/paging /vagrant_data/data/bindings /vagrant_data/data/journal /vagrant_data/data/large-messages true 2 10 10M 16000 4096 5000 90 true 120000 60000 HALT true true ACTIVEMQ.CLUSTER.ADMIN.USER XxXxX artemis-1S true artemis-1M tcp://artemis-1M:61616 tcp://artemis-1S:61616 tcp://0.0.0.0:61616 DLQ ExpiryQueue 0 -1 10 PAGE true true true true DLQ ExpiryQueue 0 -1 10 PAGE true true true true 1000
-------------------------------- Here logs from the master (artemis-1M) 2019-06-11 09:03:20,804 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server 2019-06-11 09:03:20,891 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/vagrant_data/data/journal,bindingsDirectory=/vagrant_data/data/bindings,largeMessagesDirectory=/vagrant_data/data/large-messages,pagingDirectory=/vagrant_data/data/paging) 2019-06-11 09:03:20,894 INFO [org.apache.activemq.artemis.core.server] AMQ221006: Waiting to obtain live lock 2019-06-11 09:03:20,901 INFO [org.apache.activemq.artemis.core.server] AMQ221054: libaio was found but the filesystem does not support AIO. Switching the configuration into NIO. Journal path: /vagrant_data/data/journal 2019-06-11 09:03:20,947 INFO [org.apache.activemq.artemis.core.server] AMQ221013: Using NIO Journal 2019-06-11 09:03:21,000 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824 2019-06-11 09:03:21,135 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE 2019-06-11 09:03:21,139 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP 2019-06-11 09:03:21,144 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ 2019-06-11 09:03:21,147 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT 2019-06-11 09:03:21,159 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE 2019-06-11 09:03:21,162 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP 2019-06-11 09:03:21,256 INFO [org.apache.activemq.artemis.core.server] AMQ221034: Waiting indefinitely to obtain live lock 2019-06-11 09:03:21,256 INFO [org.apache.activemq.artemis.core.server] AMQ221035: Live Server Obtained live lock 2019-06-11 09:03:22,872 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST] 2019-06-11 09:03:22,873 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ 2019-06-11 09:03:22,874 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST] 2019-06-11 09:03:22,875 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue 2019-06-11 09:03:23,230 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE] 2019-06-11 09:03:23,231 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live 2019-06-11 09:03:23,231 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.9.0 [artemis-1M, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2] 2019-06-11 09:03:23,790 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin 2019-06-11 09:03:23,970 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin 2019-06-11 09:03:24,847 INFO [io.hawt.HawtioContextListener] Initialising hawtio services 2019-06-11 09:03:24,953 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties 2019-06-11 09:03:24,956 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.5 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-) 2019-06-11 09:03:24,983 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /var/lib/artemis/tmp/uploads 2019-06-11 09:03:25,066 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal" 2019-06-11 09:03:25,160 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/var/lib/artemis/etc/jolokia-access.xml] 2019-06-11 09:03:25,203 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control 2019-06-11 09:03:25,458 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.0.2.15, 192.168.1.10, artemis-1M] 2019-06-11 09:03:25,934 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://192.168.1.10:8161 2019-06-11 09:03:25,934 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://192.168.1.10:8161/console/jolokia 2019-06-11 09:03:25,935 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://192.168.1.10:8161/console 2019-06-11 09:03:40,771 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=c4b47250, clientID=null, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5ad64273[ID=c4b47250, local= /192.168.1.10:61616, remote=/192.168.1.20:46298]] 2019-06-11 09:03:43,865 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=bc326c4b, clientID=null, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5779cb07[ID=bc326c4b, local= /192.168.1.10:61616, remote=/192.168.1.20:46300]] 2019-06-11 09:03:50,102 INFO [io.hawt.web.LoginServlet] hawtio login is using 1800 sec. HttpSession timeout 2019-06-11 09:04:46,618 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=11f124a7, clientID=null, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3714b817[ID=11f124a7, local= /192.168.1.10:61616, remote=/192.168.1.1:49368]] 2019-06-11 09:04:46,677 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a24539f0-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,701 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a24539f0-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,720 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a25147e1-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,743 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a25147e1-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,748 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a255dbc2-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,754 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a255dbc2-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,763 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a257b083-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,768 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a257b083-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,778 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a259ac54-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,784 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a259ac54-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,793 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a25c4465-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,807 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a25c4465-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,819 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a25fede6-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,829 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a25fede6-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,839 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a2637057-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,844 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a2637057-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,855 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a2659338-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,865 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a2659338-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,871 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a2687969-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,878 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a2687969-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,887 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a26ac35a-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,899 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue: QueueImpl[name=jms.artemis-test-queue, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=46af3acc-7654-11e9-a97e-08002741bcc2], temp=false]@5377bc42 2019-06-11 09:04:46,909 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a26ac35a-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,942 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a26ff37b-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:46,963 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a26ff37b-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:46,999 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a27c287c-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:47,011 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a27c287c-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:47,020 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a27f0ead-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:47,023 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a27f0ead-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:47,033 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a281589e-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:47,040 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a281589e-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:47,054 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a283f0af-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:47,058 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a283f0af-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:47,064 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a285eb80-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:47,071 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a285eb80-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:47,077 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a2880e61-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:47,085 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a2880e61-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:47,090 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a289e322-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:47,094 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a289e322-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:47,112 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a28d1773-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:47,117 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a28d1773-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:47,128 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a28ec524-8c62-11e9-b465-0a0027000007, session connectionID: 11f124a7 2019-06-11 09:04:47,134 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a28ec524-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:08:54,948 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a27f0ead-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:54,954 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a257b083-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:54,959 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a27c287c-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:54,964 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a26ff37b-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:54,974 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a289e322-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:54,980 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a28ec524-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:54,984 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a2687969-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:54,988 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a25147e1-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:54,991 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a259ac54-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:54,995 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a2659338-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,002 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a26ac35a-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,006 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a281589e-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,010 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a25fede6-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,013 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a2880e61-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,017 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a283f0af-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,020 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a255dbc2-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,025 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a25c4465-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,029 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a2637057-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,033 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a285eb80-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,057 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841008: destroyed queue: QueueImpl[name=jms.artemis-test-queue, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=46af3acc-7654-11e9-a97e-08002741bcc2], temp=false]@5377bc42, with args address: jms.artemis-test-queue, session: null, checkConsumerCount: true, removeConsumers: false, autoDeleteAddress: true 2019-06-11 09:08:55,071 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a28d1773-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,077 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a24539f0-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,085 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed connection: RemotingConnectionImpl [ID=11f124a7, clientID=null, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3714b817[ID=11f124a7, local= /192.168.1.10:61616, remote=/192.168.1.1:49368]] -------------------------------- Here logs from the slave (artemis-1S) 2019-06-11 09:03:39,069 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server 2019-06-11 09:03:39,186 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/vagrant_data/data/journal,bindingsDirectory=/vagrant_data/data/bindings,largeMessagesDirectory=/vagrant_data/data/large-messages,pagingDirectory=/vagrant_data/data/paging) 2019-06-11 09:03:39,206 INFO [org.apache.activemq.artemis.core.server] AMQ221032: Waiting to become backup node 2019-06-11 09:03:39,213 INFO [org.apache.activemq.artemis.core.server] AMQ221033: ** got backup lock 2019-06-11 09:03:39,229 INFO [org.apache.activemq.artemis.core.server] AMQ221054: libaio was found but the filesystem does not support AIO. Switching the configuration into NIO. Journal path: /vagrant_data/data/journal 2019-06-11 09:03:39,305 INFO [org.apache.activemq.artemis.core.server] AMQ221013: Using NIO Journal 2019-06-11 09:03:39,580 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824 2019-06-11 09:03:39,800 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE 2019-06-11 09:03:39,804 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP 2019-06-11 09:03:39,810 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ 2019-06-11 09:03:39,817 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT 2019-06-11 09:03:39,819 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE 2019-06-11 09:03:39,822 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP 2019-06-11 09:03:39,997 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.9.0 [46af3acc-7654-11e9-a97e-08002741bcc2] started, waiting live to fail before it gets active 2019-06-11 09:03:40,413 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin 2019-06-11 09:03:40,917 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin 2019-06-11 09:03:42,338 INFO [io.hawt.HawtioContextListener] Initialising hawtio services 2019-06-11 09:03:42,467 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties 2019-06-11 09:03:42,470 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.5 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-) 2019-06-11 09:03:42,537 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /var/lib/artemis/tmp/uploads 2019-06-11 09:03:42,625 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal" 2019-06-11 09:03:42,700 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/var/lib/artemis/etc/jolokia-access.xml] 2019-06-11 09:03:42,741 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control 2019-06-11 09:03:42,972 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.0.2.15, 192.168.1.20, artemis-1S] 2019-06-11 09:03:43,205 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST] 2019-06-11 09:03:43,207 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ 2019-06-11 09:03:43,208 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST] 2019-06-11 09:03:43,208 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue 2019-06-11 09:03:43,637 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://192.168.1.20:8161 2019-06-11 09:03:43,638 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://192.168.1.20:8161/console/jolokia 2019-06-11 09:03:43,638 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://192.168.1.20:8161/console 2019-06-11 09:03:43,948 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE] 2019-06-11 09:03:43,949 INFO [org.apache.activemq.artemis.core.server] AMQ221010: Backup Server is now live 2019-06-11 09:03:44,153 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=1b099a88, clientID=null, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5a8ff24f[ID=1b099a88, local= /192.168.1.20:61616, remote=/192.168.1.10:32771]] 2019-06-11 09:03:44,153 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=f2c1179f, clientID=null, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5471a473[ID=f2c1179f, local= /192.168.1.20:61616, remote=/192.168.1.10:32770]] 2019-06-11 09:03:52,414 INFO [io.hawt.web.LoginServlet] hawtio login is using 1800 sec. HttpSession timeout 2019-06-11 09:04:56,745 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=4669a445, clientID=null, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@95c6e1a[ID=4669a445, local= /192.168.1.20:61616, remote=/192.168.1.1:49370]] 2019-06-11 09:04:56,795 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a84c7f27-8c62-11e9-b465-0a0027000007, session connectionID: 4669a445 2019-06-11 09:04:56,807 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a84c7f27-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:56,816 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: a855a6e8-8c62-11e9-b465-0a0027000007, session connectionID: 4669a445 2019-06-11 09:04:56,821 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : a855a6e8-8c62-11e9-b465-0a0027000007, key: jms-session, data: 2019-06-11 09:04:56,869 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue: QueueImpl[name=jms.artemis-test-queue, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=46af3acc-7654-11e9-a97e-08002741bcc2], temp=false]@5afd2bff 2019-06-11 09:08:34,440 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a855a6e8-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,088 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: a84c7f27-8c62-11e9-b465-0a0027000007, failed: false 2019-06-11 09:08:55,142 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed connection: RemotingConnectionImpl [ID=4669a445, clientID=null, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@95c6e1a[ID=4669a445, local= /192.168.1.20:61616, remote=/192.168.1.1:49370]] ------------------- I'm also experiencing issues with the file system, like below. This log is coming from the master which has never been stopped (so it should be supposed to have the lock on FS, right?): 2019-06-11 09:14:46,128 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=7968f6d9, clientID=null, nodeID=46af3acc-7654-11e9-a97e-08002741bcc2, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3929b5fe[ID=7968f6d9, local= /192.168.1.10:61616, remote=/192.168.1.1:65294]] 2019-06-11 09:14:46,134 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: 079934a6-8c64-11e9-ba9a-0a0027000007, session connectionID: 7968f6d9 2019-06-11 09:14:46,137 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : 079934a6-8c64-11e9-ba9a-0a0027000007, key: jms-session, data: 2019-06-11 09:14:46,147 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: 079b3077-8c64-11e9-ba9a-0a0027000007, session connectionID: 7968f6d9 2019-06-11 09:14:46,150 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841004: added session metadata for session name : 079b3077-8c64-11e9-ba9a-0a0027000007, key: jms-session, data: 2019-06-11 09:14:46,169 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue: QueueImpl[name=jms.artemis-test-queue, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=46af3acc-7654-11e9-a97e-08002741bcc2], temp=false]@6fe0e942 2019-06-11 09:14:48,824 WARN [org.apache.activemq.artemis.journal] AMQ142009: Deleting JournalFileImpl: (activemq-data-235.amq id = 361, recordID = 361) as it does not have the configured size 2019-06-11 09:14:48,839 WARN [org.apache.activemq.artemis.journal] AMQ142009: Deleting JournalFileImpl: (activemq-data-362.amq id = 362, recordID = 362) as it does not have the configured size 2019-06-11 09:14:48,844 WARN [org.apache.activemq.artemis.journal] AMQ142009: Deleting JournalFileImpl: (activemq-data-363.amq id = 363, recordID = 363) as it does not have the configured size 2019-06-11 09:14:48,848 WARN [org.apache.activemq.artemis.journal] AMQ142009: Deleting JournalFileImpl: (activemq-data-364.amq id = 364, recordID = 364) as it does not have the configured size 2019-06-11 09:18:21,919 ERROR [org.apache.activemq.artemis.journal] AMQ144002: Error pushing opened file: ActiveMQIOErrorException[errorType=IO_ERROR message=AMQ149000: failed to rename file activemq-data-432.amq.tmp to activemq-data-432.amq] at org.apache.activemq.artemis.core.io.AbstractSequentialFile.renameTo(AbstractSequentialFile.java:161) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile0(JournalFilesRepository.java:633) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:574) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.takeFile(JournalFilesRepository.java:535) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.pushOpenedFile(JournalFilesRepository.java:486) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository$1.run(JournalFilesRepository.java:92) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.9.0.jar:2.9.0] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.9.0.jar:2.9.0] 2019-06-11 09:18:21,926 WARN [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=NULL, message=unable to open : ActiveMQIOErrorException[errorType=IO_ERROR message=AMQ149000: failed to rename file activemq-data-432.amq.tmp to activemq-data-432.amq] at org.apache.activemq.artemis.core.io.AbstractSequentialFile.renameTo(AbstractSequentialFile.java:161) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile0(JournalFilesRepository.java:633) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:574) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.takeFile(JournalFilesRepository.java:535) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.pushOpenedFile(JournalFilesRepository.java:486) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository$1.run(JournalFilesRepository.java:92) [artemis-journal-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.9.0.jar:2.9.0] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.9.0.jar:2.9.0] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.9.0.jar:2.9.0] 2019-06-11 09:18:21,948 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: 079934a6-8c64-11e9-ba9a-0a0027000007, failed: true 2019-06-11 09:18:21,949 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841003: closed session with session name: 079b3077-8c64-11e9-ba9a-0a0027000007, failed: true 2019-06-11 09:18:22,049 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter 2019-06-11 09:18:22,051 INFO [io.hawt.HawtioContextListener] Destroying hawtio services 2019-06-11 09:18:22,117 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin 2019-06-11 09:18:22,122 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin 2019-06-11 09:18:22,157 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.9.0 [46af3acc-7654-11e9-a97e-08002741bcc2] stopped, uptime 15 minutes -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html