Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@activemq.apache.org
Message-ID: <702506716.1262046760480.JavaMail.jira@brutus.apache.org>
Date: Tue, 29 Dec 2009 00:32:40 +0000 (UTC)
From: "Stirling Chow (JIRA)" <jira@apache.org>
To: dev@activemq.apache.org
Subject: [jira] Updated: (AMQ-2552) Automatic message expiration may not
 occur on a queue with no (or slow) consumers.
In-Reply-To: <804640514.1262046160535.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/activemq/browse/AMQ-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stirling Chow updated AMQ-2552:
-------------------------------

    Description: 
Symptom
========
If a queue is loaded with 400+ messages that never expire, and then a message that does expire is added to the end of the queue, the added message will not be processed by the Broker's expiration logic until some of the 400+ unexpired messages are consumed.  A unit test is provided to demonstrate this issue.

These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.

Motivation
========
Our application uses message expiration to trigger actions when messages are left in a queue for too long a period.  In many cases, there are no consumers on the queue containing the messages that might expire.  Prior versions of AMQ only expired messages when an attempt was made to dispatch them to a consumer, but AMQ-1112 introduce the notion of periodic, automatic expiration of pending messages (see the expireMessagesPeriod policy).  With this periodic expiration, our expectation is that expired messages will be detacted and processed in a timely fashion, regardless of the presence of consumers on the queue.

Cause
======
AMQ-2481 removed forced paging from the periodically-called expireMessages() method:

>>
        doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
<<
        doBrowse(browsedMessages, this.getMaxExpirePageSize());

The purpose of the change was to limit the number of messages that would be paged into memory from the message store.  The side effect is that only the first 400 messages in the queue are ever inspected by expireMessages().  If these 400 messages do not contain the messages that are expiring and there are no active consumers to move the expiring messages forward, then expiration will never occur.  With AMQ 5.3, successive calls to expireMessages() would eventually pull all messages into the page so they could be checked for expiration.

Resolution
=========
I don't think the browse idiom works very well for the automated expiration task.  It was my initial assumption (and expectation) that the expireMessagesPeriod task would process *all* expired messages in the queue *each time* the task fired.  A related issue to limiting the number of messages that are examined during each expiration task is that expiring messages at the end of the queue will not be handled in a timely fashion since they must wait for multiple firings of the expiration task (this is on 5.3 code; 5.3.1 code will never get to those messages).

If the desire/need is to scan all messages to determine which have expired, the existing paging strategy is limited because it would require all messages to be brought from store into memory, thus defeating the whole point of paging.  An alternative might be to have a sliding page window that would allow batches of messages to be brought in while clearing out the previous batch.

Another solution might be to have the expiration task page in directly from the store, but this is problematic since non-persistent messages would still have to come in from the pending message cursor.  In addition, a complete scan of the store might be non-performant for large queues.

I think the most satisfactory solution for minimizing unecessary processing and providing timely expiration of messages is to implement a proper heap that indexes messages with expiration values and allows immediate access to *only* the messages that have expired.  The references to these expired messages could then be used to remove the messages from the cursors and the store.


  was:
Symptom
========
If a queue is loaded with 400+ messages that never expire, and then a message that does expire is added to the end of the queue, the added message will not be processed by the Broker's expiration logic until some of the 400+ unexpired messages are consumed.  A unit test is provided to demonstrate this issue.

These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.

Motivation
========
Our application uses message expiration to trigger actions when messages are left in a queue for too long a period.  In many cases, there are no consumers on the queue containing the messages that might expire.  Prior versions of AMQ only expired messages when an attempt was made to dispatch them to a consumer, but AMQ-1112 introduce the notion of periodic, automatic expiration of pending messages (see the expireMessagesPeriod policy).  With this periodic expiration, our expectation is that expired messages will be detacted and processed in a timely fashion, regardless of the presence of consumers on the queue.

Cause
======
AMQ-2481 removed forced paging from the periodically-called expireMessages() method:

Resolution
=========


     Regression: [Regression, Broken Unit Test]

> Automatic message expiration may not occur on a queue with no (or slow) consumers.
> ----------------------------------------------------------------------------------
>
>                 Key: AMQ-2552
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2552
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.3.1
>         Environment: ActiveMQ 5.3.1 Snapshot r893661
>            Reporter: Stirling Chow
>         Attachments: AMQTest.java
>
>
> Symptom
> ========
> If a queue is loaded with 400+ messages that never expire, and then a message that does expire is added to the end of the queue, the added message will not be processed by the Broker's expiration logic until some of the 400+ unexpired messages are consumed.  A unit test is provided to demonstrate this issue.
> These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.
> Motivation
> ========
> Our application uses message expiration to trigger actions when messages are left in a queue for too long a period.  In many cases, there are no consumers on the queue containing the messages that might expire.  Prior versions of AMQ only expired messages when an attempt was made to dispatch them to a consumer, but AMQ-1112 introduce the notion of periodic, automatic expiration of pending messages (see the expireMessagesPeriod policy).  With this periodic expiration, our expectation is that expired messages will be detacted and processed in a timely fashion, regardless of the presence of consumers on the queue.
> Cause
> ======
> AMQ-2481 removed forced paging from the periodically-called expireMessages() method:
> >>
>         doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
> <<
>         doBrowse(browsedMessages, this.getMaxExpirePageSize());
> The purpose of the change was to limit the number of messages that would be paged into memory from the message store.  The side effect is that only the first 400 messages in the queue are ever inspected by expireMessages().  If these 400 messages do not contain the messages that are expiring and there are no active consumers to move the expiring messages forward, then expiration will never occur.  With AMQ 5.3, successive calls to expireMessages() would eventually pull all messages into the page so they could be checked for expiration.
> Resolution
> =========
> I don't think the browse idiom works very well for the automated expiration task.  It was my initial assumption (and expectation) that the expireMessagesPeriod task would process *all* expired messages in the queue *each time* the task fired.  A related issue to limiting the number of messages that are examined during each expiration task is that expiring messages at the end of the queue will not be handled in a timely fashion since they must wait for multiple firings of the expiration task (this is on 5.3 code; 5.3.1 code will never get to those messages).
> If the desire/need is to scan all messages to determine which have expired, the existing paging strategy is limited because it would require all messages to be brought from store into memory, thus defeating the whole point of paging.  An alternative might be to have a sliding page window that would allow batches of messages to be brought in while clearing out the previous batch.
> Another solution might be to have the expiration task page in directly from the store, but this is problematic since non-persistent messages would still have to come in from the pending message cursor.  In addition, a complete scan of the store might be non-performant for large queues.
> I think the most satisfactory solution for minimizing unecessary processing and providing timely expiration of messages is to implement a proper heap that indexes messages with expiration values and allows immediate access to *only* the messages that have expired.  The references to these expired messages could then be used to remove the messages from the cursors and the store.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.