From dev-return-14542-apmail-activemq-dev-archive=activemq.apache.org@activemq.apache.org Mon Mar 02 21:42:23 2009 Return-Path: Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: (qmail 97415 invoked from network); 2 Mar 2009 21:42:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 2 Mar 2009 21:42:23 -0000 Received: (qmail 34839 invoked by uid 500); 2 Mar 2009 21:42:23 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 34821 invoked by uid 500); 2 Mar 2009 21:42:23 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 34810 invoked by uid 99); 2 Mar 2009 21:42:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 13:42:22 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 21:42:20 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 8B7F9234C4AE for ; Mon, 2 Mar 2009 13:41:59 -0800 (PST) Message-ID: <437561955.1236030119570.JavaMail.jira@brutus> Date: Mon, 2 Mar 2009 13:41:59 -0800 (PST) From: "ying (JIRA)" To: dev@activemq.apache.org Subject: [jira] Commented: (AMQ-2102) Master/slave out of sync with multiple consumers In-Reply-To: <1569221231.1234277580141.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/activemq/browse/AMQ-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=50156#action_50156 ] ying commented on AMQ-2102: --------------------------- Definitely a major issue. I just run into this pre-production and this needs to be fixed. > Master/slave out of sync with multiple consumers > ------------------------------------------------ > > Key: AMQ-2102 > URL: https://issues.apache.org/activemq/browse/AMQ-2102 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 5.2.0 > Reporter: Dan James > Attachments: master.xml, MasterSlaveBug.java, slave.xml > > > I'm seeing exceptions like this in a simple master/slave setup: > ERROR Service - Async error occurred: javax.jms.JMSException: Slave broker out of sync with master: Dispatched message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the pending list for MasterSlaveBug > javax.jms.JMSException: Slave broker out of sync with master: Dispatched message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the pending list for MasterSlaveBug > The problem only happens when there are multiple consumers listening to the queue, and is more likely to occur as there are more consumers listening. I've written a test program that demonstrates the problem. > I start the master and slave with an empty data directory and let them both startup and settle. Then start the test program. The test program creates a specified number of consumers, and then starts queuing 256 messages. The consumers process the message by sending a reply. The producer counts the replies. Both consumers and the producer see all the messages, but with multiple consumers it is very likely that the error above will occur and several of the messages will still be queued on the slave. > While debugging through the activemq code, I noticed that both the master and the slave dispatch the message to a consumer's pending list independently. In other words, it is possible that the master will add the message to consumer A's pending list and the slave will add the message to consumer B's pending list. Once the message has been processed by consumer A, the master sends a message to the slaving which specifies consumer A so that the slave can remove the message. The slave looks on its copy of consumer A's pending list and cannot find the message. As a result, it throws this exception and the message stays stuck on consumer B's pending list on the slave. > Master and slave configurations along with MasterSlaveBug.java are attached to this issue. > Start master and slave brokers: > activemq xbean:master.xml > activemq xbean:slave.xml > Run with (only one consumer, the bug does not appear): > java -classpath .:activemq-all-5.2.0.jar MasterSlaveBug 1 > Run with (sixteen consumers, the bug does appear): > java -classpath .:activemq-all-5.2.0.jar MasterSlaveBug 16 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.