activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mike_vs <mvanst...@mismi.com>
Subject Ver 5.3 Pure MasterSlave does not work - sync issue
Date Wed, 17 Mar 2010 21:22:12 GMT


Version:	ActiveMq 5.3 from Apache
OS:		Linux Ubuntu
Clients:	C++ and JAVA

Problem:	ActiveMq Clustering Pure MasterSlave Problems - Slave broker out of
sync with master
                                                                Error
occured while processing sync command:
                                                                
Using:		queues and PERSISTENT TOPICS with multiple listeners to PERSISTENT
TOPICS.

Load:		queues max 200,000 messages/day, topics 2,000,000 messages/day to < 8
listeners.


Help please
I am trying to use the ActiveMq  Pure Master Slave  mechanism ofhigh
availability.
I have two boxes/machines: BOX1 has a MASTER Broker, and BOX2 has the SLAVE
broker.


MASTER started at 00:05 AM on BOX1
SLAVE  started at 00:15 AM on BOX2

CLIENTS STARTED after 02:00AM


ActiveMQ restarted every day, all ./data journals are removed (deleted) from
file system before starting
so every day is "new/clean".

I am certain this is a simple problem, however I do not know the "magic"
configuration
lines to make PureMasterSlave work. 




==================================================================
I continue to see this error in the MASTER LOG FILE


2010-03-17 14:01:09,438 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore |
ActiveMQ Transport: tcp:///10.128.34.55:2501
2010-03-17 14:01:09,438 | ERROR | Slave Failed |
org.apache.activemq.broker.ft.MasterBroker | ActiveMQ 
Transport: tcp:///10.128.34.55:2501
javax.jms.JMSException: Slave broker out of sync with master: Dispatched
message 
(444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:3698) was not in the pending list
for 
41435b44-7a26-f46a-9ff3-ba46d8fef6a5:0:0 on PUB
LIC
        at
org.apache.activemq.broker.region.PrefetchSubscription.processMessageDispatchNotification(PrefetchSubscription.java:183)
        at
org.apache.activemq.broker.region.AbstractRegion.processDispatchNotification(AbstractRegion.java:412)
        at
org.apache.activemq.broker.region.RegionBroker.processDispatchNotification(RegionBroker.java:593)
        at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at
org.apache.activemq.broker.MutableBrokerFilter.processDispatchNotification(MutableBrokerFilter.java:209)
        at
org.apache.activemq.broker.TransportConnection.processMessageDispatchNotification(TransportConnection.java:458)
        at
org.apache.activemq.command.MessageDispatchNotification.visit(MessageDispatchNotification.java:77)
        at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
        at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
        at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)
        at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
        at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
        
2010-03-17 14:01:09,439 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore |
ActiveMQ Transport: tcp:///10.128.34.55:2501
2010-03-17 14:01:09,439 | ERROR | Slave Failed |
org.apache.activemq.broker.ft.MasterBroker | ActiveMQ 
Transport: tcp:///10.128.34.55:2501
javax.jms.JMSException: Slave broker out of sync with master: Dispatched
message 
(444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:3698) was not in the pending list
for 
46494190-1d95-869d-7bd7-c1ae1182d2b9:0:0 on PUB
LIC
        at
org.apache.activemq.broker.region.PrefetchSubscription.processMessageDispatchNotification(PrefetchSubscription.java:183)
        at
org.apache.activemq.broker.region.AbstractRegion.processDispatchNotification(AbstractRegion.java:412)
        at
org.apache.activemq.broker.region.RegionBroker.processDispatchNotification(RegionBroker.java:593)
        at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at
org.apache.activemq.broker.MutableBrokerFilter.processDispatchNotification(MutableBrokerFilter.java:209)
        at
org.apache.activemq.broker.TransportConnection.processMessageDispatchNotification(TransportConnection.java:458)
        at
org.apache.activemq.command.MessageDispatchNotification.visit(MessageDispatchNotification.java:77)
        at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
        at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
        at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)
        at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
        at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2010-03-17 14:01:09,439 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore | ActiveMQ Transport:
tcp:///10.128.34.55:2501
2010-03-17 14:01:09,439 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore | ActiveMQ Transport:
tcp:///10.128.34.55:2501
2010-03-17 14:01:09,439 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore | ActiveMQ Transport:
tcp:///10.128.34.55:2501


==================================================
and these errors in the SLAVE LOG FILE

2010-03-17 14:20:13,361 | DEBUG | adding message
ref:444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11430 |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,361 | DEBUG | Batch update done. |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,361 | DEBUG | Doing batch update... adding: 1 removing:
0  | org.apache.activemq.store.amq.AMQMessageStore | Checkpoint:
topic://PUBLIC
2010-03-17 14:20:13,362 | DEBUG | adding message
ref:6d404d3a-8f81-c648-0519-ec9512d668ae:0:0:10529 |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,362 | DEBUG | Batch update done. |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC

2010-03-17 14:20:13,362 | DEBUG | Journalled message add for:
44458305-ee21-cca9-c582-2c26e14962ad:0:0:1921, at: offset = 23462708, file =
1, size = 355, type = 1 
| org.apache.activemq.store.amq.AMQMessageStore | VMTransport: vm://null#1
2010-03-17 14:20:13,362 | DEBUG | Doing batch update... adding: 1 removing:
0  | org.apache.activemq.store.amq.AMQMessageStore | Checkpoint:
topic://PUBLIC
2010-03-17 14:20:13,362 | DEBUG | adding message
ref:44458305-ee21-cca9-c582-2c26e14962ad:0:0:1921 |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,362 | DEBUG | Batch update done. |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,363 | DEBUG | Error occured while processing sync
command: MessageAck {commandId = 965169, responseRequired = true, ackType =
2, 
consumerId = ID:nirrti-47630-1268845079477-0:7:1:1, firstMessageId =
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429, lastMessageId =
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429,
destination = topic://PUBLIC, transactionId = null, messageCount = 1},
exception: javax.jms.JMSException: Unmatched acknowledege: MessageAck
{commandId = 965169, responseRequired = true, ackType = 2, consumerId =
ID:nirrti-47630-1268845079477-0:7:1:1, firstMessageId =
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429, lastMessageId =
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429, destination =
topic://PUBLIC, transactionId = null, messageCount = 1}; Could not find
Message-ID 444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429 in dispatched-list
(start of ack) | org.apache.activemq.broker.TransportConnection.Service |
VMTransport: vm://null#1
javax.jms.JMSException: Unmatched acknowledege: MessageAck {commandId =
965169, responseRequired = true, ackType = 2, consumerId =
ID:nirrti-47630-1268845079477-0:7:1:1, 
firstMessageId = 444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429,
lastMessageId = 444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429, destination
= topic://PUBLIC,
transactionId = null, messageCount = 1}; Could not find Message-ID
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429 in dispatched-list (start of
ack)
        at
org.apache.activemq.broker.region.PrefetchSubscription.assertAckMatchesDispatched(PrefetchSubscription.java:440)
        at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:208)
        at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:369)
        at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:470)
        at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
        at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
        at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
        at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
        at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:449)
        at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
        at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
        at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
        at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)
        at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
        at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2010-03-17 14:20:13,363 | DEBUG | FixFromAlgoOm toPageIn: 1, Inflight: 0,
pagedInMessages.size 0 | org.apache.activemq.broker.region.Queue |
VMTransport: vm://null#1
2010-03-17 14:20:13,364 | DEBUG | Error occured while processing sync
command: MessageDispatchNotification {commandId = 965175, 
responseRequired = true, consumerId =
41435b44-7a26-f46a-9ff3-ba46d8fef6a5:0:0, destination = topic://PUBLIC,
messageId = 444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11430,
deliverySequenceId = 0}, exception: javax.jms.JMSException:
Slave broker out of sync with master: Dispatched message
(444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11430) was not in the 
pending list for 41435b44-7a26-f46a-9ff3-ba46d8fef6a5:0:0 on PUBLIC |
org.apache.activemq.broker.TransportConnection.Service | VMTransport:
vm://null#1
javax.jms.JMSException: Slave broker out of sync with master: Dispatched
message (444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11430) was 
not in the pending list for 41435b44-7a26-f46a-9ff3-ba46d8fef6a5:0:0 on
PUBLIC
        at
org.apache.activemq.broker.region.PrefetchSubscription.processMessageDispatchNotification(PrefetchSubscription.java:183)
        at
org.apache.activemq.broker.region.AbstractRegion.processDispatchNotification(AbstractRegion.java:412)
        at
org.apache.activemq.broker.region.RegionBroker.processDispatchNotification(RegionBroker.java:593)
        at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at
org.apache.activemq.broker.MutableBrokerFilter.processDispatchNotification(MutableBrokerFilter.java:209)
        at
org.apache.activemq.broker.TransportConnection.processMessageDispatchNotification(TransportConnection.java:458)
        at
org.apache.activemq.command.MessageDispatchNotification.visit(MessageDispatchNotification.java:77)
        at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)



======================================================================================
MASTER xml file


<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.apache.org/schema/core"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">

    <!-- Allows us to use system properties as variables in this
configuration file -->
    <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.base}/conf/credentials.properties</value>
        </property>
    </bean>


    <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data">

        <managementContext>
            <managementContext createConnector="true"/>
        </managementContext>

        <persistenceAdapter>
            <amqPersistenceAdapter directory="${activemq.base}/data/amqpa"     
maxFileLength="32mb"/>
        </persistenceAdapter>

        <destinationPolicy>
            <policyMap>
              <policyEntries>

                <policyEntry topic=">" producerFlowControl="false"
memoryLimit="300mb">

                    <dispatchPolicy>
                        <strictOrderDispatchPolicy />
                    </dispatchPolicy>
                </policyEntry>

                <policyEntry queue=">" producerFlowControl="false"
memoryLimit="300mb">

                    <dispatchPolicy>
                        <strictOrderDispatchPolicy />
                    </dispatchPolicy>

                </policyEntry>


              </policyEntries>
            </policyMap>
        </destinationPolicy>


        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="1000 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="10 gb" name="storeUsage10GB"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="1000 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <transportConnectors>
            <transportConnector name="openwire"
uri="tcp://0.0.0.0:61616?transport.KeepAliveResponseRequired=true"/>
        </transportConnectors>

    </broker>
    <import resource="jetty.xml"/>
</beans>







======================================================================================
SLAVE XML FILE


<!--
-->
<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.apache.org/schema/core"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">

    <!-- Allows us to use system properties as variables in this
configuration file -->
    <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.base}/conf/credentials.properties</value>
        </property>
    </bean>


    <broker
                masterConnectorURI="tcp://nirrti:61616"
                shutdownOnMasterFailure="false"
                xmlns="http://activemq.apache.org/schema/core"
                brokerName="activemq_SLAVE"
                dataDirectory="${activemq.base}/data" >

        <managementContext>
            <managementContext createConnector="true"/>
        </managementContext>

        <persistenceAdapter>
            <amqPersistenceAdapter directory="${activemq.base}/data/amqpa"     
maxFileLength="32mb"/>
        </persistenceAdapter>

        <destinationPolicy>
            <policyMap>
              <policyEntries>


                <policyEntry topic=">" producerFlowControl="false"
memoryLimit="300mb">

                    <dispatchPolicy>
                        <strictOrderDispatchPolicy />
                    </dispatchPolicy>

                </policyEntry>


                <policyEntry queue=">" producerFlowControl="false"
memoryLimit="100mb">

                    <dispatchPolicy>
                        <strictOrderDispatchPolicy />
                    </dispatchPolicy>

                </policyEntry>

              </policyEntries>
            </policyMap>
        </destinationPolicy>


        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="1000 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="10 gb" name="storage10GB"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="1000 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>


        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
        </transportConnectors>


    </broker>

    <import resource="jetty.xml"/>
</beans>



-- 
View this message in context: http://old.nabble.com/Ver-5.3-Pure-MasterSlave-does-not-work---sync-issue-tp27937803p27937803.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message