Mailing-List: contact issues-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@activemq.apache.org
Date: Mon, 7 Dec 2015 20:13:10 +0000 (UTC)
From: "Christopher L. Shannon (JIRA)" <jira@apache.org>
To: issues@activemq.apache.org
Message-ID: <JIRA.12614294.1351725616000.293387.1449519190967@Atlassian.JIRA>
In-Reply-To: <JIRA.12614294.1351725616000@Atlassian.JIRA>
References: <JIRA.12614294.1351725616000@Atlassian.JIRA>
 <JIRA.12614294.1351725616856@arcas>
Subject: [jira] [Closed] (AMQ-4151) Duplicate non-persistent messages that
 are sent to a queue are either dispatched multiple times (i.e., not
 detected as duplicates) or cause the queue size to be miscalculated.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/AMQ-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Christopher L. Shannon closed AMQ-4151.
---------------------------------------
    Resolution: Incomplete

no test case provided

> Duplicate non-persistent messages that are sent to a queue are either dispatched multiple times (i.e., not detected as duplicates) or cause the queue size to be miscalculated.
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: AMQ-4151
>                 URL: https://issues.apache.org/jira/browse/AMQ-4151
>             Project: ActiveMQ
>          Issue Type: Bug
>            Reporter: Stirling Chow
>
> Symptom
> =======
> We have a virtual topic that is shared by a network of brokers.  Each broker has consumers that process the corresponding Consumer.*.VirtualTopic.> queue.
> While testing the effects of memory limits on our system, we encountered AMQ-4148, which resulted in duplicate subscriptions being made to the virtual topic.  These duplicate subcriptions were attempting to enqueue the same topic message multiple times to the corresponding Consumer.*.VirtualTopic.>.
> Although logic exists in queues to handle duplicate messages, we noticed lots of strange behaviour with duplicate handling.  We decided to investigate further and found a race condition whereby duplicate non-persistent messages that were correctly ignored resulted in the queue size appearing to be non-empty when in fact there were no messages.
> While our original investigation was prompted by AMQ-4148, the test case we attached to this ticket does not rely on the misbehaviour caused by AMQ-4148, but can occur during normal operation of network bridges when there are multiple consumers for a virtual topic (as is the case, e.g., when conduit subscriptions are disabled).
> Cause
> =====
> If a virtual topic has multiple consumers, then sending a message to the topic results in multiple dispatches of the same message (one to each consumer).  If the multiple consumers originate from the same remote broker (e.g., because conduit subscriptions are disabled), then a single message sent to the virtual topic on broker1 will result in multiple duplicate messages being sent to the virtual topic on broker2.
> If broker2 has corresponding Consumer.*.VirtualTopic.> queue, then the multiple duplicate messages sent to the virtual topic on broker2 will result in multiple duplicate messages being sent to the queue.  
> If the messages are non-persistent, the only logic that prevents each duplicate message from being dispatched to a consumer is as follows:
> {code:title=Queue.java}
> private PendingList doPageInForDispatch(boolean force) throws Exception {
> ...
>         // Only add new messages, not already pagedIn to avoid multiple
>         // dispatch attempts
>         pagedInMessagesLock.writeLock().lock();
>         try {
>             if(isPrioritizedMessages()) {
>                 resultList = new PrioritizedPendingList();
>             } else {
>                 resultList = new OrderedPendingList();
>             }
>             for (QueueMessageReference ref : result) {
>                 if (!pagedInMessages.containsKey(ref.getMessageId())) {
>                     pagedInMessages.put(ref.getMessageId(), ref);
>                     resultList.addMessageLast(ref);
>                 } else {
>                     ref.decrementReferenceCount();
>                 }
>             }
>         } finally {
>             pagedInMessagesLock.writeLock().unlock();
>         }
> ...        
> {code}
> If the consumers are fast and acknowledge the initial message dispatch before the next duplicate message is sent to the queue, then the {{pagedInMessages.constainsKey(...)}} check will *not* prevent the duplicate message from being dispatched since the message ID will have already been removed as part of the acknowledgement.
> This is problem 1: duplicate detection fails if acknowledgements are quick
> If the consumers are slow and don't acknowledge the initial message dispatch before the next duplicate message is sent to the queue, then duplicate detection will correctly ignore the message.  However, by this time, the duplicate message has already incremented the queue size:
> {code:title=Queue.java}
> final void messageSent(final ConnectionContext context, final Message msg) throws Exception {
>     destinationStatistics.getEnqueues().increment();
>     destinationStatistics.getMessages().increment();
>     messageDelivered(context, msg);
> {code}
> The call to {{messageSent}} is made by the thread sending the message to the queue.  This thread is completely unaware that the message was ignored since the call to {{doPageInForDispatch}} is done by a separate thread (e.g., the taskrunner calling {{iterate()}}.
> Since the queue size is incremented, but the duplicate message is never dispatched, there is no subsequent acknowledgement ot reduce the queue size.  As a result, each duplicate message that is ignored remains counted in the queue size even though it's no longer in the queue.  
> This is problem 2: if duplication detection succeeds, the queue size incorrectly counts the duplicate/ignored message.
> Which problem gets exhibited depends on a race condition between the thread(s) enqueueing the duplicate virtual topic subscriptions and consumer threads acknowledging the dispatches.
> While the unit test's methodology of creating multiple virtual topic consumers is somewhat contrived, there must be other circumstances under which duplicate messages are sent to queues (otherwise, why would there be logic to handle this case?)  In this context, the test case reveals a problem with the handling of duplicate messages.
> Solution
> ========
> None at this time.  We worked around the problem by patching AMQ-4148 to prevent the duplicate subscriptions that cause this bug.  However, our concern has been raised about AMQ's generally handling of duplicate messages sent to a queue.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)