activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tca...@cac.washington.edu
Subject Re: pure master/slave problems (ActiveMQ v1.1.1)
Date Mon, 18 Jun 2007 21:57:53 GMT

Opps, I meant ActiveMQ v4.1.1.

On Mon, 18 Jun 2007, tcaine@cac.washington.edu wrote:

>
> I'm trying to run ActiveMQ v1.1.1 in a pure master/slave setup on a pair of 
> RHEL servers using Java(TM) 2 Runtime Environment, Standard Edition (build 
> 1.5.0_11-b03) and it doesn't seem to reliably handle failover.
>
> The first issue I am seeing is that the master server keeps generating the 
> following message in the log file:
>
> 2007-06-18 10:04:22,961 [127.0.0.1:52327] ERROR MasterBroker - Slave Failed
> java.lang.AssertionError: Unsupported Method
> at 
> org.apache.activemq.transport.TransportSupport.request(TransportSupport.java:71)
> at 
> org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
> at 
> org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
> at 
> org.apache.activemq.transport.MutexTransport.request(MutexTransport.java:54)
> at 
> org.apache.activemq.broker.ft.MasterBroker.sendSyncToSlave(MasterBroker.java:363)
> at 
> org.apache.activemq.broker.ft.MasterBroker.sendToSlave(MasterBroker.java:345)
> at 
> org.apache.activemq.broker.ft.MasterBroker.acknowledge(MasterBroker.java:320)
> at 
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:88)
> at 
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:506)
> at org.apache.activemq.command.MessageAck.visit(MessageAck.java:179)
> at 
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> at 
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
> at 
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
> at 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
> at 
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
> at 
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
> at java.lang.Thread.run(Thread.java:595)
>
> I'm not sure if this is a problem, but it doesn't give my any warm fuzzies. 
> Can this be ignored or is this a real problem?
>
> After the slave starts up the connection between the master and the slave 
> times out (from the perspective of the slave), the slave tries to start up 
> it's connectors and fails.  Here's what I'm seeing in the slave's log file:
>
> 2007-06-18 10:04:11,735 [main           ] INFO  BrokerService - ActiveMQ 
> 4.1.1 JMS Message Broker (shakira-slave) is starting
> 2007-06-18 10:04:11,845 [JMX connector  ] INFO  ManagementContext - JMX 
> consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
> 2007-06-18 10:04:12,721 [main           ] INFO  JDBCPersistenceAdapter - 
> Database driver recognized: [apache_derby_embedded_jdbc_driver]
> 2007-06-18 10:04:13,943 [main           ] INFO  DefaultDatabaseLocker - 
> Attempting to acquire the exclusive lock to become the Master broker
> 2007-06-18 10:04:13,944 [main           ] INFO  DefaultDatabaseLocker - 
> Becoming the master on dataSource: 
> org.apache.derby.jdbc.EmbeddedDataSource@1add463
> 2007-06-18 10:04:14,012 [main           ] INFO  JournalPersistenceAdapter - 
> Journal Recovery Started from: Active Journal: using 5 x 20.0 Megs at: 
> /data/run/activemq-data/journal
> 2007-06-18 10:04:14,079 [main           ] INFO  JournalPersistenceAdapter - 
> Journal Recovered: 0 message(s) in transactions recovered.
> 2007-06-18 10:04:14,221 [main           ] INFO  TransportServerThreadSupport 
> - Listening for connections at: tcp://slave.cac.washington.edu:61616
> 2007-06-18 10:04:14,235 [main           ] INFO  TransportConnector - 
> Connector openwire Started
> 2007-06-18 10:04:14,244 [main           ] INFO  TransportServerThreadSupport 
> - Listening for connections at: stomp://slave.cac.washington.edu:61613
> 2007-06-18 10:04:14,244 [main           ] INFO  TransportConnector - 
> Connector stomp Started
> 2007-06-18 10:04:14,247 [main           ] INFO  NetworkConnector - Network 
> Connector default-nc Started
> 2007-06-18 10:04:14,253 [main           ] INFO  TransportConnector - 
> Connector vm://shakira-slave Started
> 2007-06-18 10:04:14,281 [main           ] INFO  MasterConnector - Starting a 
> network connection between vm://shakira-slave#0 and tcp://null:0 has been 
> established.
> 2007-06-18 10:04:14,287 [main           ] INFO  BrokerService - ActiveMQ JMS 
> Message Broker (shakira-slave, 
> ID:slave.cac.washington.edu-38591-1182186251386-2:0) started
> 2007-06-18 10:04:14,333 [Thread-4       ] INFO  MasterConnector - Slave 
> connection between vm://shakira-slave#0 and 
> tcp://master.cac.washington.edu/10.10.10.1:61616 has been established.
> 2007-06-18 10:04:14,740 [ Agent Notifier] INFO  NetworkConnector - 
> Establishing network connection between from vm://shakira-slave?network=true 
> to tcp://master.cac.washington.edu:61616
> 2007-06-18 10:04:14,762 [Thread-7       ] INFO  DemandForwardingBridge - 
> Network connection between vm://shakira-slave#2 and 
> tcp://master.cac.washington.edu/10.10.10.1:61616(shakira-master) has been 
> established.
> 2007-06-18 12:52:44,648 [iveMQ Scheduler] ERROR MasterConnector - Network 
> connection between vm://shakira-slave#0 and 
> tcp://master.cac.washington.edu/10.10.10.1:61616 shutdown: Channel was 
> inactive for too long.
> 2007-06-18 12:52:44,771 [iveMQ Scheduler] WARN  BrokerService - Master Failed 
> - starting all connectors
> 2007-06-18 12:52:44,773 [iveMQ Scheduler] ERROR BrokerService - Failed to 
> startAllConnectors
>
> The slave keeps timing out and then fails to startup the connectors.  I have 
> no idea why the channel is inactive since there are thousands of events per 
> minute being sent into ActiveMQ topics using the a failover URL of:
>
> failover://(tcp://master.cac.washington.edu:61616,tcp://slave.cac.washington.edu:61616)?randomize=false
>
> If I kill the master server (kill -9) the slave doesn't take over at this 
> point.  The slave doesn't seem to log any messages once startAllConnectors 
> fails, although the activeMQ process is still running.
>
> I've also tried restarting the slave once the slave's connection to the 
> master broker times out and the master server seems to think the slave is 
> still connected and won't let the slave reconnect.
>
> My master config is:
>
> <beans>
>
> <bean 
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
>
> <broker brokerName="shakira-master" useJmx="true" 
> xmlns="http://activemq.org/config/1.0">
>
> <memoryManager>
>    <usageManager id="memory-manager" limit="1 GB"/>
> </memoryManager>
>
> <persistenceAdapter>
>    <journaledJDBC journalLogFiles="5" 
> dataDirectory="/data/run/activemq-data"/>
> </persistenceAdapter>
>
> <transportConnectors>
>    <transportConnector name="openwire" 
> uri="tcp://master.cac.washington.edu:61616" 
> discoveryUri="multicast://default"/>
>    <transportConnector name="stomp" 
> uri="stomp://master.cac.washington.edu:61613"/>
> </transportConnectors>
>
> <networkConnectors>
>    <networkConnector name="default-nc" uri="multicast://default"/>
> </networkConnectors>
>
> </broker>
>
> </beans>
>
> And my slave config is:
>
> <beans>
>
> <bean 
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
>
> <broker brokerName="shakira-slave" useJmx="true" 
> masterConnectorURI="tcp://master.cac.washington.edu:61616" 
> shutdownOnMasterFailure="false" xmlns="http://activemq.org/config/1.0">
>
> <memoryManager>
>    <usageManager id="memory-manager" limit="1 GB"/>
> </memoryManager>
>
> <persistenceAdapter>
>    <journaledJDBC journalLogFiles="5" 
> dataDirectory="/data/run/activemq-data"/>
> </persistenceAdapter>
>
> <transportConnectors>
>    <transportConnector name="openwire" 
> uri="tcp://slave.cac.washington.edu:61616" 
> discoveryUri="multicast://default"/>
>    <transportConnector name="stomp" 
> uri="stomp://slave.cac.washington.edu:61613"/>
> </transportConnectors>
>
> <networkConnectors>
>    <networkConnector name="default-nc" uri="multicast://default"/>
> </networkConnectors>
>
> </broker>
>
> </beans>
>
> Has anyone had the same problem?  Our product is getting ready to be released 
> and the reliablility problems with the pure master/slave setup is becoming a 
> show stopper for us.  Any help would be greatly appreciated.
>
> Thanks,
> Todd
>

Mime
View raw message