activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stirling Chow (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMQ-4151) Duplicate non-persistent messages that are sent to a queue are either dispatched multiple times (i.e., not detected as duplicates) or cause the queue size to be miscalculated.
Date Wed, 31 Oct 2012 23:21:11 GMT
Stirling Chow created AMQ-4151:
----------------------------------

             Summary: Duplicate non-persistent messages that are sent to a queue are either
dispatched multiple times (i.e., not detected as duplicates) or cause the queue size to be
miscalculated.
                 Key: AMQ-4151
                 URL: https://issues.apache.org/jira/browse/AMQ-4151
             Project: ActiveMQ
          Issue Type: Bug
            Reporter: Stirling Chow


Symptom
=======
We have a virtual topic that is shared by a network of brokers.  Each broker has consumers
that process the corresponding Consumer.*.VirtualTopic.> queue.

While testing the effects of memory limits on our system, we encountered AMQ-4148, which resulted
in duplicate subscriptions being made to the virtual topic.  These duplicate subcriptions
were attempting to enqueue the same topic message multiple times to the corresponding Consumer.*.VirtualTopic.>.

Although logic exists in queues to handle duplicate messages, we noticed lots of strange behaviour
with duplicate handling.  We decided to investigate further and found a race condition whereby
duplicate non-persistent messages that were correctly ignored resulted in the queue size appearing
to be non-empty when in fact there were no messages.

While our original investigation was prompted by AMQ-4148, the test case we attached to this
ticket does not rely on the misbehaviour caused by AMQ-4148, but can occur during normal operation
of network bridges when there are multiple consumers for a virtual topic (as is the case,
e.g., when conduit subscriptions are disabled).

Cause
=====
If a virtual topic has multiple consumers, then sending a message to the topic results in
multiple dispatches of the same message (one to each consumer).  If the multiple consumers
originate from the same remote broker (e.g., because conduit subscriptions are disabled),
then a single message sent to the virtual topic on broker1 will result in multiple duplicate
messages being sent to the virtual topic on broker2.

If broker2 has corresponding Consumer.*.VirtualTopic.> queue, then the multiple duplicate
messages sent to the virtual topic on broker2 will result in multiple duplicate messages being
sent to the queue.  

If the messages are non-persistent, the only logic that prevents each duplicate message from
being dispatched to a consumer is as follows:

{code:title=Queue.java}
private PendingList doPageInForDispatch(boolean force) throws Exception {
...
        // Only add new messages, not already pagedIn to avoid multiple
        // dispatch attempts
        pagedInMessagesLock.writeLock().lock();
        try {
            if(isPrioritizedMessages()) {
                resultList = new PrioritizedPendingList();
            } else {
                resultList = new OrderedPendingList();
            }
            for (QueueMessageReference ref : result) {
                if (!pagedInMessages.containsKey(ref.getMessageId())) {
                    pagedInMessages.put(ref.getMessageId(), ref);
                    resultList.addMessageLast(ref);
                } else {
                    ref.decrementReferenceCount();
                }
            }
        } finally {
            pagedInMessagesLock.writeLock().unlock();
        }
...        
{code}

If the consumers are fast and acknowledge the initial message dispatch before the next duplicate
message is sent to the queue, then the {{pagedInMessages.constainsKey(...)}} check will *not*
prevent the duplicate message from being dispatched since the message ID will have already
been removed as part of the acknowledgement.

This is problem 1: duplicate detection fails if acknowledgements are quick

If the consumers are slow and don't acknowledge the initial message dispatch before the next
duplicate message is sent to the queue, then duplicate detection will correctly ignore the
message.  However, by this time, the duplicate message has already incremented the queue size:

{code:title=Queue.java}
final void messageSent(final ConnectionContext context, final Message msg) throws Exception
{
    destinationStatistics.getEnqueues().increment();
    destinationStatistics.getMessages().increment();
    messageDelivered(context, msg);
{code}

The call to {{messageSent}} is made by the thread sending the message to the queue.  This
thread is completely unaware that the message was ignored since the call to {{doPageInForDispatch}}
is done by a separate thread (e.g., the taskrunner calling {{iterate()}}.

Since the queue size is incremented, but the duplicate message is never dispatched, there
is no subsequent acknowledgement ot reduce the queue size.  As a result, each duplicate message
that is ignored remains counted in the queue size even though it's no longer in the queue.
 

This is problem 2: if duplication detection succeeds, the queue size incorrectly counts the
duplicate/ignored message.

Which problem gets exhibited depends on a race condition between the thread(s) enqueueing
the duplicate virtual topic subscriptions and consumer threads acknowledging the dispatches.

While the unit test's methodology of creating multiple virtual topic consumers is somewhat
contrived, there must be other circumstances under which duplicate messages are sent to queues
(otherwise, why would there be logic to handle this case?)  In this context, the test case
reveals a problem with the handling of duplicate messages.

Solution
========
None at this time.  We worked around the problem by patching AMQ-4148 to prevent the duplicate
subscriptions that cause this bug.  However, our concern has been raised about AMQ's generally
handling of duplicate messages sent to a queue.





--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message