activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From marc2112 <marc2...@yahoo.com>
Subject topology recommendation?
Date Tue, 31 Mar 2009 18:05:52 GMT
Hey Folks,

Can you chime in with any thoughts on how you'd configure ActiveMQ for this architecture...?

I have about 10 instances of a web application that require the services of an external component.
 The external component is written in C/C++ and uses the CMS library to connect to ActiveMQ
5.2.  The web applications run in Jetty 6 on top of a Java 6 JVM.

Every web server session is assigned to an instance of the external component.
Every web server instance creates an exclusive topic on which messages destined for their
end users are to be placed.  A correlation ID consisting of username and sessionid are attached
to every message.

Every C/C++ service subscribes to a common Queue for connection requests.  The Queue has a
roundRobinDispatchPolicy and a noSubscriptionRecoveryPolicy configuration.  When the service
receives a connect request, it replies with a unique topic that further commands for that
correlationID should be sent to.  This topic is unique to an instance of the service.

High availability is required for the architecture such that in the event an ActiveMQ server
dies, the application can continue to run.  We can tolerate very occasional message loss especially
if it buys us a performance benefit.

We need to support about 15k msg/s across the system and need to be able to scale beyond that.

Given these parameters, I thought it would make sense to configure two ActiveMQ brokers in
a network-of-brokers configuration.  This is the part of the activemq.xml file that I have
for that:
           <!-- by default just auto discover the other brokers -->
            <networkConnector uri="multicast://default?group=${activemq.ENV}"
                name="default-nc"
                networkTTL="2"
                dynamicOnly="true"
                conduitSubscriptions="true"
                decreaseNetworkConsumerPriority="false">
            </networkConnector>

Does this look right?  Everything works well until we get up to about 10k msgs/s across the
system.   After this point, the connections between the two brokers aborted and clients across
the system had to reconnect causing large CPU spikes.

The webserver connections were pretty evenly distributed across the two JMS servers.  The
C/C++ components were almost all connected to only one of the JMS servers.  It's this server
that was showing increased CPU utilization before the two brokers stopped talking to each
other momentarily.  Could that be an issue?

Here's the exception that I saw on server2 (the one with all of the C/C++ connections):
2009-03-31 12:18:49,426 [rt: vm://null#6] WARN  DemandForwardingBridge         - Caught an
exception processing local command
java.net.SocketException: Socket closed
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
        at org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
        at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
        at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
        at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
        at org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:647)
        at org.apache.activemq.network.DemandForwardingBridgeSupport$1.onCommand(DemandForwardingBridgeSupport.java:147)
        at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2009-03-31 12:18:49,430 [ Agent Notifier] INFO  DemandForwardingBridge         - amqbroker_jms2
bridge to amqbroker_jms1 stopped
2009-03-31 12:18:49,750 [ Agent Notifier] INFO  DiscoveryNetworkConnector      - Establishing
network connection between from vm://null to tcp://jms1.radiusim.com:61516
2009-03-31 12:18:49,750 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms2 instead
2009-03-31 12:18:49,750 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms2 instead
2009-03-31 12:18:49,750 [ Agent Notifier] INFO  TransportConnector             - Connector
vm://null Started
2009-03-31 12:19:02,545 [ker=vm://null#8] INFO  DemandForwardingBridge         - Network connection
between vm://null#8 and tcp://jms1.radiusim.com/10.2.20.1:61516(amqbroker_jms1) has
 been established.
2009-03-31 12:19:24,049 [ Agent Notifier] INFO  DemandForwardingBridge         - amqbroker_jms2
bridge to amqbroker_jms1 stopped
2009-03-31 12:19:24,049 [rt: vm://null#8] WARN  DemandForwardingBridge         - Caught an
exception processing local command
java.net.SocketException: Socket closed
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)

Here's what I saw on server1:
2009-03-31 12:19:12,637 [ Agent Notifier] INFO  DemandForwardingBridge         - amqbroker_jms1
bridge to amqbroker_jms2 stopped
2009-03-31 12:19:15,914 [ Agent Notifier] INFO  DiscoveryNetworkConnector      - Establishing
network connection between from vm://null to tcp://jms2.mycompany.com:61516
2009-03-31 12:19:15,916 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms1 instead
2009-03-31 12:19:15,916 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms1 instead
2009-03-31 12:19:15,916 [ Agent Notifier] INFO  TransportConnector             - Connector
vm://null Started
2009-03-31 12:19:22,118 [ Agent Notifier] INFO  DemandForwardingBridge         - amqbroker_jms1
bridge to Unknown stopped
2009-03-31 12:19:23,621 [ Agent Notifier] INFO  DiscoveryNetworkConnector      - Establishing
network connection between from vm://null to tcp://jms2.mycompany.com:61516
2009-03-31 12:19:23,623 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms1 instead
2009-03-31 12:19:23,623 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms1 instead
2009-03-31 12:19:23,623 [ Agent Notifier] INFO  TransportConnector             - Connector
vm://null Started
2009-03-31 12:19:33,740 [ Agent Notifier] INFO  DemandForwardingBridge         - amqbroker_jms1
bridge to Unknown stopped
2009-03-31 12:19:33,740 [ Agent Notifier] INFO  DiscoveryNetworkConnector      - Establishing
network connection between from vm://null to tcp://jms2.mycompany.com:61516
2009-03-31 12:19:33,741 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms1 instead
2009-03-31 12:19:33,741 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms1 instead
2009-03-31 12:19:33,741 [ Agent Notifier] INFO  TransportConnector             - Connector
vm://null Started
2009-03-31 12:19:44,031 [er=vm://null#22] WARN  DemandForwardingBridge         - Network connection
between vm://null#22 and tcp://jms2.mycompany.com/192.168.1.4:61516 shutdown due to a remo
te error: java.io.IOException: Wire format negotiation timeout: peer did not send his wire
format.
2009-03-31 12:19:44,166 [NetworkBridge  ] INFO  DemandForwardingBridge         - amqbroker_jms1
bridge to Unknown stopped
2009-03-31 12:19:49,169 [ Agent Notifier] INFO  DiscoveryNetworkConnector      - Establishing
network connection between from vm://null to tcp://jms2.mycompany.com:61516
2009-03-31 12:19:49,170 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms1 instead
2009-03-31 12:19:49,170 [ Agent Notifier] WARN  BrokerRegistry                 - Broker localhost
not started so using amqbroker_jms1 instead
2009-03-31 12:19:49,170 [ Agent Notifier] INFO  TransportConnector             - Connector
vm://null Started
2009-03-31 12:19:59,398 [er=vm://null#24] WARN  DemandForwardingBridge         - Network connection
between vm://null#24 and tcp://jms2.mycompany.com/192.168.1.4:61516 shutdown due to a remo
te error: java.io.IOException: Wire format negotiation timeout: peer did not send his wire
format.



Other configuration I have:
       <destinationPolicy>
            <policyMap>
                <policyEntries>
                        <!-- Configure the Interop connection dispatch queue -->
                        <policyEntry queue="Queue.com.mycompany.connectrequests" memoryLimit="5mb">
                                <dispatchPolicy>
                                        <roundRobinDispatchPolicy/>
                                </dispatchPolicy>
                                <subscriptionRecoveryPolicy>
                            <noSubscriptionRecoveryPolicy/>
                        </subscriptionRecoveryPolicy>
                        </policyEntry>
                    <policyEntry queue=">" memoryLimit="8mb"/>
                    <policyEntry topic=">" memoryLimit="8mb"/>
                </policyEntries>
            </policyMap>
        </destinationPolicy>

        <!-- Use the following to configure how ActiveMQ is exposed in JMX -->
        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>


        <persistenceAdapter>
            <amqPersistenceAdapter syncOnWrite="false" directory="${activemq.base}/data"
maxFileLength="20 mb"/>
        </persistenceAdapter>

        <sslContext>
            <sslContext keyStore="file:${activemq.base}/conf/broker.ks" keyStorePassword="password"
trustStore="file:${activemq.base}/conf/broker.ts" trustStorePassword="password"/>
        </sslContext>

        <!--  The maximum about of space the broker will use before slowing down producers
-->
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="128 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="1 gb" name="foo"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="128 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>


        <!-- The transport connectors ActiveMQ will listen to -->
        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://localhost:61516" discoveryUri="multicast://default?group=${activemq.ENV}"/>
        </transportConnectors>


      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message