Return-Path: 
 <dev-return-11615-apmail-activemq-dev-archive=activemq.apache.org@activemq.apache.org>
Delivered-To: apmail-activemq-dev-archive@www.apache.org
Received: (qmail 96072 invoked from network); 21 Jul 2008 07:30:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 21 Jul 2008 07:30:00 -0000
Received: (qmail 73825 invoked by uid 500); 21 Jul 2008 07:30:00 -0000
Delivered-To: apmail-activemq-dev-archive@activemq.apache.org
Received: (qmail 73807 invoked by uid 500); 21 Jul 2008 07:30:00 -0000
Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@activemq.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@activemq.apache.org>
List-Post: <mailto:dev@activemq.apache.org>
List-Id: <dev.activemq.apache.org>
Reply-To: dev@activemq.apache.org
Delivered-To: mailing list dev@activemq.apache.org
Received: (qmail 73796 invoked by uid 99); 21 Jul 2008 07:30:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jul 2008 00:30:00 -0700
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jul 2008 07:29:05 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 75902234C173
	for <dev@activemq.apache.org>; Mon, 21 Jul 2008 00:29:00 -0700 (PDT)
Message-ID: <855922325.1216625340480.JavaMail.jira@brutus>
Date: Mon, 21 Jul 2008 00:29:00 -0700 (PDT)
From: "Bryan Shaw (JIRA)" <jira@apache.org>
To: dev@activemq.apache.org
Subject: [jira] Issue Comment Edited: (AMQ-1845) Message loss in network of
 brokers when network connection break
In-Reply-To: <1989902634.1215524101182.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/activemq/browse/AMQ-1845?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D44=
408#action_44408 ]=20

silverhoof edited comment on AMQ-1845 at 7/21/08 12:27 AM:
-----------------------------------------------------------

This test can reproduce the message loss using ActiveMQ 5.0 and 5.1.

The producer side have its own broker and have a fixed IP address.
On the producer side, the spring configuration is like follows:
{code:xml}
<?xml version=3D"1.0" encoding=3D"UTF-8"?>
<beans xmlns=3D"http://www.springframework.org/schema/beans" xmlns:xsi=3D"h=
ttp://www.w3.org/2001/XMLSchema-instance"
=09xmlns:jee=3D"http://www.springframework.org/schema/jee" xmlns:amq=3D"htt=
p://activemq.apache.org/schema/core"
=09xsi:schemaLocation=3D"http://www.springframework.org/schema/beans http:/=
/www.springframework.org/schema/beans/spring-beans-2.0.xsd
=09http://www.springframework.org/schema/jee http://www.springframework.org=
/schema/jee/spring-jee-2.0.xsd
=09http://activemq.apache.org/schema/core http://activemq.apache.org/schema=
/core/activemq-core.xsd">

=09<amq:broker brokerName=3D"producer" useJmx=3D"true" persistent=3D"true">
=09=09<amq:transportConnectors>
=09=09=09<amq:transportConnector uri=3D"tcp://localhost:61616" />
=09=09</amq:transportConnectors>
=09</amq:broker>

=09<!-- Jms ConnectionFactory -->
=09<bean id=3D"jmsFactory" class=3D"org.apache.activemq.ActiveMQConnectionF=
actory">
=09=09<property name=3D"brokerURL" value=3D"vm://localhost:61616" />
=09</bean>


=09<!-- Spring JMS SimpleConverter -->
=09<bean id=3D"simpleConverter" class=3D"org.springframework.jms.support.co=
nverter.SimpleMessageConverter" />

=09<!-- JMS Queue Template -->
=09<bean id=3D"jmsTemplate" class=3D"org.springframework.jms.core.JmsTempla=
te">
=09=09<property name=3D"connectionFactory" ref=3D"jmsFactory" />
=09=09<property name=3D"messageConverter" ref=3D"converter" />
=09</bean>

=09<!-- Message Converter -->
=09<bean id=3D"converter" class=3D"com.al.SimpleConverter">
=09=09<property name=3D"converter">
=09=09=09<ref local=3D"simpleConverter" />
=09=09</property>
=09</bean>

=09<!-- Message porducer -->
=09<bean id=3D"sender" class=3D"com.al.DefaultSender">
=09=09<property name=3D"template" ref=3D"jmsTemplate" />
=09=09<property name=3D"destinationName" value=3D"test-out" />
=09</bean>

</beans>
{code}

The test app at producer side:
{code:title=3DTestApp.java|borderStyle=3Dsolid}
package com.al;

import org.apache.log4j.Logger;

public class TestApp {
=09/**
=09 * Logger for this class
=09 */
=09private static final Logger logger =3D Logger.getLogger(TestApp.class);

=09public static void main(String[] args) {

=09=09logger.debug("start test...");
=09=09//Initializing spring context
=09=09Context.init();
=09=09// uncomment to send messages

=09=09DefaultSender sender =3D Context.getBean("sender");
=09=09int idx =3D 1;
=09=09int count =3D 3000;
=09=09while (idx <=3D count) {
=09=09=09sender.sendMessage(SimpleMessageHelper.genSimpleMessage(idx));
=09=09=09logger.debug("send out message : payload is " + idx);
=09=09=09idx++;
=09=09}

=09=09/* Infinitely hold main thread  to keep the spring context running
=09=09*/
=09=09Object lock =3D new Object();
=09=09synchronized (lock) {
=09=09=09try {
=09=09=09=09lock.wait();
=09=09=09} catch (InterruptedException e) {
=09=09=09=09e.printStackTrace();
=09=09=09}
=09=09}

=09}
}
{code}

The test client is very simple which send 3000 messages to the queue "test-=
out" on the producer side broker.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

The consumer side using dynamic IP address and deployed on another machine.
We have a broker on one separate machine using a duplex =3D "true" network =
connector
Consumer side Spring configuration file is as follows:
{code:xml}
<?xml version=3D"1.0" encoding=3D"UTF-8"?>
<beans xmlns=3D"http://www.springframework.org/schema/beans" xmlns:xsi=3D"h=
ttp://www.w3.org/2001/XMLSchema-instance"
=09xmlns:jee=3D"http://www.springframework.org/schema/jee" xmlns:amq=3D"htt=
p://activemq.apache.org/schema/core"
=09xsi:schemaLocation=3D"http://www.springframework.org/schema/beans http:/=
/www.springframework.org/schema/beans/spring-beans-2.0.xsd
=09http://www.springframework.org/schema/jee http://www.springframework.org=
/schema/jee/spring-jee-2.0.xsd
=09http://activemq.apache.org/schema/core http://activemq.apache.org/schema=
/core/activemq-core.xsd">

=09<amq:broker brokerName=3D"brokerA" useJmx=3D"true" persistent=3D"true">

=09=09<amq:networkConnectors>
=09=09=09<amq:networkConnector uri=3D"static://(tcp://=E3=80=90ip-for-produ=
cer-machine=E3=80=91:61616)" duplex=3D"true"/>
=09=09</amq:networkConnectors>

=09=09<amq:transportConnectors>
=09=09=09<amq:transportConnector uri=3D"tcp://localhost:61616" />
=09=09</amq:transportConnectors>

=09</amq:broker>

=09<!-- Jms ConnectionFactory -->

=09<bean id=3D"jmsFactory" class=3D"org.apache.activemq.ActiveMQConnectionF=
actory">
=09=09<property name=3D"brokerURL" value=3D"vm://localhost:61616" />
=09</bean>

=09<!-- Spring JMS SimpleConverter -->
=09<bean id=3D"simpleConverter" class=3D"org.springframework.jms.support.co=
nverter.SimpleMessageConverter" />


=09<!-- Message Converter -->
=09<bean id=3D"converter" class=3D"com.al.SimpleConverter">
=09=09<property name=3D"converter">
=09=09=09<ref local=3D"simpleConverter" />
=09=09</property>
=09</bean>

=09<!-- MDP -->
=09<!-- consumer 1 -->
=09<bean id=3D"listener" class=3D"org.springframework.jms.listener.adapter.=
MessageListenerAdapter">
=09=09<constructor-arg>
=09=09=09<bean class=3D"com.al.DefaultListener" />
=09=09</constructor-arg>
=09=09<property name=3D"defaultListenerMethod" value=3D"onMessage" />
=09=09<property name=3D"messageConverter" ref=3D"converter" />
=09</bean>

=09<bean id=3D"listenerContainer" class=3D"org.springframework.jms.listener=
.DefaultMessageListenerContainer">
=09=09<property name=3D"connectionFactory" ref=3D"jmsFactory" />
=09=09<property name=3D"destinationName" value=3D"test-out" />
=09=09<property name=3D"messageListener" ref=3D"listener" />
=09=09<property name=3D"sessionTransacted" value=3D"true" />
=09</bean>
</beans>
{code}

Default Listener code:
{code:title=3DDefaultListener.java|borderStyle=3Dsolid}
package com.al;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

public class DefaultListener {

=09private static Log logger =3D LogFactory.getLog(DefaultListener.class);

=09static int i =3D 1;

=09public void onMessage(SimpleMessage message) {
=09=09int j =3D Integer.valueOf(message.getPayLoad());
=09=09logger.debug("receive : j =3D " + j);
=09=09if (i !=3D j) {
=09=09=09logger.debug("warn : i=3D" + i + ",j=3D" + j);
=09=09}
=09=09i++;
=09}
}
{code}

The default listener print out the message and compare the index in the mes=
sage with the number of message received (It should match if no message los=
s occurs).


=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
And this is how to reproduce message loss in this environment.
1. Start producer application and consumer application
2. Producer application starts to send 3000 messages to the queue "test-out=
"
3. Consumer application starts to receive messages from the local distribut=
ed queue.
4. During the message sending process, disconnect the network cable on cons=
umer machine.
5. You will find that the producer machine still sending messages to the qu=
eue "test-out", which is normal because these messages should be persistent=
 in producer broker's "test-out" queue. And the mean time the consumer side=
 stopped getting messages from the "test-out" queue, which is also normal.
6. When you connect the network cable again on the consumer machine(during =
the message sending or after the producer node finished sending message) , =
the consumer broker receives message from the "test-out" queue again. But l=
ogger displays the message count miss match with the message index in the m=
essage payload, which indicates message loss happened.


And by the way, this also reproduced when we trying to avoid duplex connect=
ion by define network connector on both side. This is archived by fixed IP =
node connecting to a dynamic IP node using a domain name (simulated by usin=
g hosts file to provide routing information)



      was (Author: silverhoof):
    This test can reproduce the message loss using ActiveMQ 5.0 and 5.1.

The producer side have its own broker and have a fixed IP address.
On the producer side, the spring configuration is like follows:
{quote}
<?xml version=3D"1.0" encoding=3D"UTF-8"?>
<beans xmlns=3D"http://www.springframework.org/schema/beans" xmlns:xsi=3D"h=
ttp://www.w3.org/2001/XMLSchema-instance"
=09xmlns:jee=3D"http://www.springframework.org/schema/jee" xmlns:amq=3D"htt=
p://activemq.apache.org/schema/core"
=09xsi:schemaLocation=3D"http://www.springframework.org/schema/beans http:/=
/www.springframework.org/schema/beans/spring-beans-2.0.xsd
=09http://www.springframework.org/schema/jee http://www.springframework.org=
/schema/jee/spring-jee-2.0.xsd
=09http://activemq.apache.org/schema/core http://activemq.apache.org/schema=
/core/activemq-core.xsd">

=09<amq:broker brokerName=3D"producer" useJmx=3D"true" persistent=3D"true">
=09=09<amq:transportConnectors>
=09=09=09<amq:transportConnector uri=3D"tcp://localhost:61616" />
=09=09</amq:transportConnectors>
=09</amq:broker>

=09<!-- Jms ConnectionFactory -->
=09<bean id=3D"jmsFactory" class=3D"org.apache.activemq.ActiveMQConnectionF=
actory">
=09=09<property name=3D"brokerURL" value=3D"vm://localhost:61616" />
=09</bean>


=09<!-- Spring JMS SimpleConverter -->
=09<bean id=3D"simpleConverter" class=3D"org.springframework.jms.support.co=
nverter.SimpleMessageConverter" />

=09<!-- JMS Queue Template -->
=09<bean id=3D"jmsTemplate" class=3D"org.springframework.jms.core.JmsTempla=
te">
=09=09<property name=3D"connectionFactory" ref=3D"jmsFactory" />
=09=09<property name=3D"messageConverter" ref=3D"converter" />
=09</bean>

=09<!-- Message Converter -->
=09<bean id=3D"converter" class=3D"com.al.SimpleConverter">
=09=09<property name=3D"converter">
=09=09=09<ref local=3D"simpleConverter" />
=09=09</property>
=09</bean>

=09<!-- Message porducer -->
=09<bean id=3D"sender" class=3D"com.al.DefaultSender">
=09=09<property name=3D"template" ref=3D"jmsTemplate" />
=09=09<property name=3D"destinationName" value=3D"test-out" />
=09</bean>

</beans>
{quote}

The test app at producer side:
{quote}
package com.al;

import org.apache.log4j.Logger;

public class TestApp {
=09/**
=09 * Logger for this class
=09 */
=09private static final Logger logger =3D Logger.getLogger(TestApp.class);

=09public static void main(String[] args) {

=09=09logger.debug("start test...");
=09=09//Initializing spring context
=09=09Context.init();
=09=09// uncomment to send messages

=09=09DefaultSender sender =3D Context.getBean("sender");
=09=09int idx =3D 1;
=09=09int count =3D 3000;
=09=09while (idx <=3D count) {
=09=09=09sender.sendMessage(SimpleMessageHelper.genSimpleMessage(idx));
=09=09=09logger.debug("send out message : payload is " + idx);
=09=09=09idx++;
=09=09}

=09=09/* Infinitely hold main thread  to keep the spring context running
=09=09*/
=09=09Object lock =3D new Object();
=09=09synchronized (lock) {
=09=09=09try {
=09=09=09=09lock.wait();
=09=09=09} catch (InterruptedException e) {
=09=09=09=09e.printStackTrace();
=09=09=09}
=09=09}

=09}
}
{quote}

The test client is very simple which send 3000 messages to the queue "test-=
out" on the producer side broker.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

The consumer side using dynamic IP address and deployed on another machine.
We have a broker on one separate machine using a duplex =3D "true" network =
connector
Consumer side Spring configuration file is as follows:
<?xml version=3D"1.0" encoding=3D"UTF-8"?>
<beans xmlns=3D"http://www.springframework.org/schema/beans" xmlns:xsi=3D"h=
ttp://www.w3.org/2001/XMLSchema-instance"
=09xmlns:jee=3D"http://www.springframework.org/schema/jee" xmlns:amq=3D"htt=
p://activemq.apache.org/schema/core"
=09xsi:schemaLocation=3D"http://www.springframework.org/schema/beans http:/=
/www.springframework.org/schema/beans/spring-beans-2.0.xsd
=09http://www.springframework.org/schema/jee http://www.springframework.org=
/schema/jee/spring-jee-2.0.xsd
=09http://activemq.apache.org/schema/core http://activemq.apache.org/schema=
/core/activemq-core.xsd">

=09<amq:broker brokerName=3D"brokerA" useJmx=3D"true" persistent=3D"true">

=09=09<amq:networkConnectors>
=09=09=09<amq:networkConnector uri=3D"static://(tcp://=E3=80=90ip-for-produ=
cer-machine=E3=80=91:61616)" duplex=3D"true"/>
=09=09</amq:networkConnectors>

=09=09<amq:transportConnectors>
=09=09=09<amq:transportConnector uri=3D"tcp://localhost:61616" />
=09=09</amq:transportConnectors>

=09</amq:broker>

=09<!-- Jms ConnectionFactory -->

=09<bean id=3D"jmsFactory" class=3D"org.apache.activemq.ActiveMQConnectionF=
actory">
=09=09<property name=3D"brokerURL" value=3D"vm://localhost:61616" />
=09</bean>

=09<!-- Spring JMS SimpleConverter -->
=09<bean id=3D"simpleConverter" class=3D"org.springframework.jms.support.co=
nverter.SimpleMessageConverter" />


=09<!-- Message Converter -->
=09<bean id=3D"converter" class=3D"com.al.SimpleConverter">
=09=09<property name=3D"converter">
=09=09=09<ref local=3D"simpleConverter" />
=09=09</property>
=09</bean>

=09<!-- MDP -->
=09<!-- consumer 1 -->
=09<bean id=3D"listener" class=3D"org.springframework.jms.listener.adapter.=
MessageListenerAdapter">
=09=09<constructor-arg>
=09=09=09<bean class=3D"com.al.DefaultListener" />
=09=09</constructor-arg>
=09=09<property name=3D"defaultListenerMethod" value=3D"onMessage" />
=09=09<property name=3D"messageConverter" ref=3D"converter" />
=09</bean>

=09<bean id=3D"listenerContainer" class=3D"org.springframework.jms.listener=
.DefaultMessageListenerContainer">
=09=09<property name=3D"connectionFactory" ref=3D"jmsFactory" />
=09=09<property name=3D"destinationName" value=3D"test-out" />
=09=09<property name=3D"messageListener" ref=3D"listener" />
=09=09<property name=3D"sessionTransacted" value=3D"true" />
=09</bean>
</beans>

Default Listener code:
package com.al;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

public class DefaultListener {

=09private static Log logger =3D LogFactory.getLog(DefaultListener.class);

=09static int i =3D 1;

=09public void onMessage(SimpleMessage message) {
=09=09int j =3D Integer.valueOf(message.getPayLoad());
=09=09logger.debug("receive : j =3D " + j);
=09=09if (i !=3D j) {
=09=09=09logger.debug("warn : i=3D" + i + ",j=3D" + j);
=09=09}
=09=09i++;
=09}
}

The default listener print out the message and compare the index in the mes=
sage with the number of message received (It should match if no message los=
s occurs).


=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
And this is how to reproduce message loss in this environment.
1. Start producer application and consumer application
2. Producer application starts to send 3000 messages to the queue "test-out=
"
3. Consumer application starts to receive messages from the local distribut=
ed queue.
4. During the message sending process, disconnect the network cable on cons=
umer machine.
5. You will find that the producer machine still sending messages to the qu=
eue "test-out", which is normal because these messages should be persistent=
 in producer broker's "test-out" queue. And the mean time the consumer side=
 stopped getting messages from the "test-out" queue, which is also normal.
6. When you connect the network cable again on the consumer machine(during =
the message sending or after the producer node finished sending message) , =
the consumer broker receives message from the "test-out" queue again. But l=
ogger displays the message count miss match with the message index in the m=
essage payload, which indicates message loss happened.


And by the way, this also reproduced when we trying to avoid duplex connect=
ion by define network connector on both side. This is archived by fixed IP =
node connecting to a dynamic IP node using a domain name (simulated by usin=
g hosts file to provide routing information)


 =20
> Message loss in network of brokers when network connection break
> ----------------------------------------------------------------
>
>                 Key: AMQ-1845
>                 URL: https://issues.apache.org/activemq/browse/AMQ-1845
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0
>         Environment: Two brokers connected via TCP with one persistent di=
stributed queue and a producer and a consumer on each broker.
>            Reporter: Bryan Shaw
>
> Producer on broker A send 2500 message on the distributed queue at broker=
 A.
> Producer B starts to receive message from distributed queue on broker B.
> During the receiving process, the network between these two brokers down =
and later brought up again.
> In this senario, we found that some messages are lost.=20
> It seems the broker A are sending message to broker B when the network is=
 down and these messages are removed from queue in broker A but never recei=
ved by broker B which causing message loss.
> Is this a bug or a configuration problem?=20
> I thought the configuration like this is the store/forward pattern which =
should ensure the message reliability in an unstable network.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.