activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "glenn.struzinski" <glenn.struzin...@oracle.com>
Subject Network of brokers with multiple worldwide data centers
Date Fri, 13 Mar 2015 13:43:08 GMT
Looking for some advice for an ActiveMQ setup that gives both stability,
redundancy, and is able to handle 10s of thousands of clients.  This is for
an mcollective deployment and will be used exclusively to execute remote
upgrades and maintenance of systems.

Initially setup a network of brokers with two MQ servers per datacenter in a
hub and spoke topology.  Systems are virtual and initially had 4cpu/16gb ram
and kahadb.  Configured with the following params: 

<broker xmlns="http://activemq.apache.org/schema/core" useJmx="true"
brokerName="hostname" dataDirectory="${activemq.base}/data"
networkConnectorStartAsync="true" schedulePeriodForDestinationPurge="60000">

<destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="false"> 
                   <pendingSubscriberPolicy>
                     <vmCursor />
                   </pendingSubscriberPolicy>
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="false" /> 
                <policyEntry queue="*.reply.>" gcInactiveDestinations="true"
inactiveTimoutBeforeGC="300000" />
              </policyEntries>
            </policyMap>
        </destinationPolicy>

Many connections like this:

<networkConnector
            name="connection_to_remotehostname"
            uri="static:(tcp://remotehostname.com:61616)"
	    dynamicOnly="true"
            userName="replication"
            password="replication"
            duplex="true"
            networkTTL="2"
            decreaseNetworkConsumerPriority="true"
           />

 <systemUsage>
    <systemUsage>
      <memoryUsage>
        <memoryUsage limit="8 gb"/>
      </memoryUsage>
      <storeUsage>
        <storeUsage limit="1 gb"/>
      </storeUsage>
      <tempUsage>
        <tempUsage limit="100 mb"/>
      </tempUsage>
    </systemUsage>
  </systemUsage>

start script memory config
ACTIVEMQ_OPTS_MEMORY="-Xms256m -Xmx16384m
-Dorg.apache.activemq.UseDedicatedTaskRunner=false"


Found that it ran fine for about 8 hours and would start to drop connections
once it got to about 50% ram utilization.

Debug showed lots of message expiry (hostnames replace with hostname and
host)

[t201] Scheduler] DEBUG Queue                          -
queue://host_mcollective.reply.hostname_25106 expiring messages ..
2015-03-13 07:37:02,023 [t201] Scheduler] DEBUG Queue                         
- queue://host_mcollective.reply.hostname_25106 expiring messages done.
2015-03-13 07:37:02,033 [t201] Scheduler] DEBUG Queue                         
- queue://host_mcollective.reply.hostname_25434 expiring messages ..
2015-03-13 07:37:02,033 [t201] Scheduler] DEBUG Queue                         
- queue://host_mcollective.reply.hostname_25434 expiring messages done.

We have now tried to switch to a master slave topology with leveldb and
shared nfs at each data center.

Network connectors defined as master slaves from teh hub node

 <networkConnector 
      name="remote site #1"
     
uri="masterslave:(tcp://host1.domain.com:61616,tcp://host2.domain.com:61616)"
      duplex="true"
      >
    </networkConnector>

memory footprint reduced

<systemUsage>
    <systemUsage>
      <memoryUsage>
        <memoryUsage limit="8 gb"/>
      </memoryUsage>
      <storeUsage>
        <storeUsage limit="1 gb"/>
      </storeUsage>
      <tempUsage>
        <tempUsage limit="100 mb"/>
      </tempUsage>
    </systemUsage>
  </systemUsage>

Both setups utilized NIO and all of the tuning that was mentioned from this
site:  

http://www.javabeat.net/deploying-activemq-for-large-numbers-of-concurrent-applications/

Any help on the best way to create a functioning system that scales and is
able to handle possible slow links  would be much appreciated.  Real world
deployment examples would be preferred.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-with-multiple-worldwide-data-centers-tp4693158.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message