activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tca...@cac.washington.edu
Subject pure master/slave problems (ActiveMQ v1.1.1)
Date Mon, 18 Jun 2007 21:56:43 GMT

I'm trying to run ActiveMQ v1.1.1 in a pure master/slave setup on a pair 
of RHEL servers using Java(TM) 2 Runtime Environment, Standard Edition 
(build 1.5.0_11-b03) and it doesn't seem to reliably handle failover.

The first issue I am seeing is that the master server keeps generating the 
following message in the log file:

2007-06-18 10:04:22,961 [127.0.0.1:52327] ERROR MasterBroker - Slave Failed
java.lang.AssertionError: Unsupported Method
at org.apache.activemq.transport.TransportSupport.request(TransportSupport.java:71)
at org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
at org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
at org.apache.activemq.transport.MutexTransport.request(MutexTransport.java:54)
at org.apache.activemq.broker.ft.MasterBroker.sendSyncToSlave(MasterBroker.java:363)
at org.apache.activemq.broker.ft.MasterBroker.sendToSlave(MasterBroker.java:345)
at org.apache.activemq.broker.ft.MasterBroker.acknowledge(MasterBroker.java:320)
at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:88)
at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:506)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:179)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
at java.lang.Thread.run(Thread.java:595)

I'm not sure if this is a problem, but it doesn't give my any warm 
fuzzies.  Can this be ignored or is this a real problem?

After the slave starts up the connection between the master and the slave 
times out (from the perspective of the slave), the slave tries to start up 
it's connectors and fails.  Here's what I'm seeing in the slave's log 
file:

2007-06-18 10:04:11,735 [main           ] INFO  BrokerService - ActiveMQ 4.1.1 JMS Message
Broker (shakira-slave) is starting
2007-06-18 10:04:11,845 [JMX connector  ] INFO  ManagementContext - JMX consoles can connect
to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
2007-06-18 10:04:12,721 [main           ] INFO  JDBCPersistenceAdapter - Database driver recognized:
[apache_derby_embedded_jdbc_driver]
2007-06-18 10:04:13,943 [main           ] INFO  DefaultDatabaseLocker - Attempting to acquire
the exclusive lock to become the Master broker
2007-06-18 10:04:13,944 [main           ] INFO  DefaultDatabaseLocker - Becoming the master
on dataSource: org.apache.derby.jdbc.EmbeddedDataSource@1add463
2007-06-18 10:04:14,012 [main           ] INFO  JournalPersistenceAdapter - Journal Recovery
Started from: Active Journal: using 5 x 20.0 Megs at: /data/run/activemq-data/journal
2007-06-18 10:04:14,079 [main           ] INFO  JournalPersistenceAdapter - Journal Recovered:
0 message(s) in transactions recovered.
2007-06-18 10:04:14,221 [main           ] INFO  TransportServerThreadSupport - Listening for
connections at: tcp://slave.cac.washington.edu:61616
2007-06-18 10:04:14,235 [main           ] INFO  TransportConnector - Connector openwire Started
2007-06-18 10:04:14,244 [main           ] INFO  TransportServerThreadSupport - Listening for
connections at: stomp://slave.cac.washington.edu:61613
2007-06-18 10:04:14,244 [main           ] INFO  TransportConnector - Connector stomp Started
2007-06-18 10:04:14,247 [main           ] INFO  NetworkConnector - Network Connector default-nc
Started
2007-06-18 10:04:14,253 [main           ] INFO  TransportConnector - Connector vm://shakira-slave
Started
2007-06-18 10:04:14,281 [main           ] INFO  MasterConnector - Starting a network connection
between vm://shakira-slave#0 and tcp://null:0 has been established.
2007-06-18 10:04:14,287 [main           ] INFO  BrokerService - ActiveMQ JMS Message Broker
(shakira-slave, ID:slave.cac.washington.edu-38591-1182186251386-2:0) started
2007-06-18 10:04:14,333 [Thread-4       ] INFO  MasterConnector - Slave connection between
vm://shakira-slave#0 and tcp://master.cac.washington.edu/10.10.10.1:61616 has been established.
2007-06-18 10:04:14,740 [ Agent Notifier] INFO  NetworkConnector - Establishing network connection
between from vm://shakira-slave?network=true to tcp://master.cac.washington.edu:61616
2007-06-18 10:04:14,762 [Thread-7       ] INFO  DemandForwardingBridge - Network connection
between vm://shakira-slave#2 and tcp://master.cac.washington.edu/10.10.10.1:61616(shakira-master)
has been established.
2007-06-18 12:52:44,648 [iveMQ Scheduler] ERROR MasterConnector - Network connection between
vm://shakira-slave#0 and tcp://master.cac.washington.edu/10.10.10.1:61616 shutdown: Channel
was inactive for too long.
2007-06-18 12:52:44,771 [iveMQ Scheduler] WARN  BrokerService - Master Failed - starting all
connectors
2007-06-18 12:52:44,773 [iveMQ Scheduler] ERROR BrokerService - Failed to startAllConnectors

The slave keeps timing out and then fails to startup the connectors.  I 
have no idea why the channel is inactive since there are thousands of 
events per minute being sent into ActiveMQ topics using the a failover URL 
of:

failover://(tcp://master.cac.washington.edu:61616,tcp://slave.cac.washington.edu:61616)?randomize=false

If I kill the master server (kill -9) the slave doesn't take over at this 
point.  The slave doesn't seem to log any messages once startAllConnectors fails, 
although the activeMQ process is still running.

I've also tried restarting the slave once the slave's connection to the 
master broker times out and the master server seems to think the slave is 
still connected and won't let the slave reconnect.

My master config is:

<beans>

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>

<broker brokerName="shakira-master" useJmx="true" xmlns="http://activemq.org/config/1.0">

<memoryManager>
     <usageManager id="memory-manager" limit="1 GB"/>
</memoryManager>

<persistenceAdapter>
     <journaledJDBC journalLogFiles="5" dataDirectory="/data/run/activemq-data"/>
</persistenceAdapter>

<transportConnectors>
     <transportConnector name="openwire" uri="tcp://master.cac.washington.edu:61616" discoveryUri="multicast://default"/>
     <transportConnector name="stomp" uri="stomp://master.cac.washington.edu:61613"/>
</transportConnectors>

<networkConnectors>
     <networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>

</broker>

</beans>

And my slave config is:

<beans>

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>

<broker brokerName="shakira-slave" useJmx="true" masterConnectorURI="tcp://master.cac.washington.edu:61616"
shutdownOnMasterFailure="false" xmlns="http://activemq.org/config/1.0">

<memoryManager>
     <usageManager id="memory-manager" limit="1 GB"/>
</memoryManager>

<persistenceAdapter>
     <journaledJDBC journalLogFiles="5" dataDirectory="/data/run/activemq-data"/>
</persistenceAdapter>

<transportConnectors>
     <transportConnector name="openwire" uri="tcp://slave.cac.washington.edu:61616" discoveryUri="multicast://default"/>
     <transportConnector name="stomp" uri="stomp://slave.cac.washington.edu:61613"/>
</transportConnectors>

<networkConnectors>
     <networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>

</broker>

</beans>

Has anyone had the same problem?  Our product is getting ready to be 
released and the reliablility problems with the pure master/slave setup is 
becoming a show stopper for us.  Any help would be greatly appreciated.

Thanks,
Todd

Mime
View raw message