From dev-return-14596-apmail-activemq-dev-archive=activemq.apache.org@activemq.apache.org Fri Mar 06 18:09:15 2009 Return-Path: Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: (qmail 90256 invoked from network); 6 Mar 2009 18:09:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 6 Mar 2009 18:09:15 -0000 Received: (qmail 47205 invoked by uid 500); 6 Mar 2009 18:09:14 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 47187 invoked by uid 500); 6 Mar 2009 18:09:14 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 47176 invoked by uid 99); 6 Mar 2009 18:09:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Mar 2009 10:09:14 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Mar 2009 18:09:06 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id A3189234C498 for ; Fri, 6 Mar 2009 10:08:44 -0800 (PST) Message-ID: <1269958302.1236362924666.JavaMail.jira@brutus> Date: Fri, 6 Mar 2009 10:08:44 -0800 (PST) From: "Aaron Riekenberg (JIRA)" To: dev@activemq.apache.org Subject: [jira] Updated: (AMQ-2149) Shared Filesystem Master Slave: missing messages In-Reply-To: <1847668314.1236362684677.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/activemq/browse/AMQ-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Riekenberg updated AMQ-2149: ---------------------------------- Description: I'm finding occasionally messages are not delivered in order in a shared filesystem master slave setup when the master fails and the slave takes over. I'm running a simple test on one physical machine where the shared filesystem is on a single disk (no SAN currently involved). I'm attaching a shell script (run_master_slave_brokers.sh) that starts a master and slave broker in the same directory, sleeps 20 seconds, kills the master, sleeps 20 seconds, starts a new slave, sleeps 20 seconds, kills the master, etc. Also attached is a small java test program (MasterSlaveTest.java) The program starts 10 JMS senders that send 75kb text messages every 25 ms to unique queues. These messages contain a sequence number header (a long). The program also starts 10 receivers (1 for each queue) that keep track of the next expected sequence number and validate each incoming sequence number. If a receiver gets an unexpected sequence number, the test program exits (System.exit(1)). Both the senders and receivers use the failover transport to connect to the broker. Messages being sent are persistent, so in theory there should be no message loss when the master fails and slave takes over. I run the script to start the brokers, then run my test program. Most times when the script kills the master and the slave is promoted, things work fine - the test program reconnects, and messages continue to be delivered in order. If I run this long enough though, eventually my test program fails just after a slave broker is promoted to master with output similar to this: Mar 6, 2009 11:58:12 AM org.apache.activemq.transport.failover.FailoverTransport doReconnect INFO: Successfully reconnected to tcp://localhost:61616 Mar 6, 2009 11:58:12 AM org.aaron.MasterSlaveTest$Receiver onMessage WARNING: test.queue.3 received 630 expected 629 This indicates the receiver for test.queue.3 received message 630 after the slave broker took over and missed message 629. This seems to happen more often when more senders and receivers are running and more queues are in use. If I run a single sender/receiver pair on 1 queue, it is very difficult to make this happen. was: I'm finding occasionally messages are not delivered in order in a shared filesystem master slave setup when the master fails and the slave takes over. I'm running a simple test on one physical machine where the shared filesystem is on a single disk (no SAN currently involved). I'm attaching a shell script (run_master_slave_brokers.sh) that starts a master and slave broker in the same directory, sleeps 20 seconds, kills the master, sleeps 20 seconds, starts a new slave, sleeps 20 seconds, kills the master, etc. Also attached is a small java test program (MasterSlaveTest.java) The program starts 10 JMS senders that send 75kb text messages every 25 ms to unique queues. These messages contain a sequence number header (a long). The program also starts 10 receivers (1 for each queue) that keep track of the next expected sequence number and validate each incoming sequence number. If a receiver gets an unexpected sequence number, the test program exits (System.exit(1)). Both the senders and receivers use the failover transport to connect to the broker. Messages being sent are persistent, so in theory there should be no message loss when the master fails and slave takes over. I run the script to start the brokers, then run my test program. Most times when the script kills the master and the slave is promoted, things work fine - the test program reconnects, and messages continue to be delivered in order. If I run this long enough though, eventually my test program fails just after a slave broker is promoted to master with output similar to this: Mar 6, 2009 11:58:12 AM org.apache.activemq.transport.failover.FailoverTransport doReconnect INFO: Successfully reconnected to tcp://localhost:61616 Mar 6, 2009 11:58:12 AM org.aaron.MasterSlaveTest$Receiver onMessage WARNING: test.queue.3 received 630 expected 629 This indicates the receiver received message 630 after the slave broker took over. This means the receiver missed message 629. This seems to happen more often when more senders and receivers are running and more queues are in use. If I run a single sender/receiver pair on 1 queue, it is very difficult to make this happen. > Shared Filesystem Master Slave: missing messages > ------------------------------------------------ > > Key: AMQ-2149 > URL: https://issues.apache.org/activemq/browse/AMQ-2149 > Project: ActiveMQ > Issue Type: Bug > Affects Versions: 5.2.0 > Environment: Ubuntu Linux 8.10 AMD64, Sun JDK 1.6.0.10 > Reporter: Aaron Riekenberg > Attachments: activemq.xml, MasterSlaveTest.java, run_master_slave_brokers.sh > > > I'm finding occasionally messages are not delivered in order in a shared filesystem master slave setup when the master fails and the slave takes over. I'm running a simple test on one physical machine where the shared filesystem is on a single disk (no SAN currently involved). > I'm attaching a shell script (run_master_slave_brokers.sh) that starts a master and slave broker in the same directory, sleeps 20 seconds, kills the master, sleeps 20 seconds, starts a new slave, sleeps 20 seconds, kills the master, etc. > Also attached is a small java test program (MasterSlaveTest.java) The program starts 10 JMS senders that send 75kb text messages every 25 ms to unique queues. These messages contain a sequence number header (a long). The program also starts 10 receivers (1 for each queue) that keep track of the next expected sequence number and validate each incoming sequence number. If a receiver gets an unexpected sequence number, the test program exits (System.exit(1)). Both the senders and receivers use the failover transport to connect to the broker. Messages being sent are persistent, so in theory there should be no message loss when the master fails and slave takes over. > I run the script to start the brokers, then run my test program. Most times when the script kills the master and the slave is promoted, things work fine - the test program reconnects, and messages continue to be delivered in order. If I run this long enough though, eventually my test program fails just after a slave broker is promoted to master with output similar to this: > Mar 6, 2009 11:58:12 AM org.apache.activemq.transport.failover.FailoverTransport doReconnect > INFO: Successfully reconnected to tcp://localhost:61616 > Mar 6, 2009 11:58:12 AM org.aaron.MasterSlaveTest$Receiver onMessage > WARNING: test.queue.3 received 630 expected 629 > This indicates the receiver for test.queue.3 received message 630 after the slave broker took over and missed message 629. > This seems to happen more often when more senders and receivers are running and more queues are in use. If I run a single sender/receiver pair on 1 queue, it is very difficult to make this happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.