activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stirling Chow (JIRA)" <j...@apache.org>
Subject [jira] Updated: (AMQ-2552) Automatic message expiration may not occur on a queue with no (or slow) consumers.
Date Tue, 29 Dec 2009 00:38:40 GMT

     [ https://issues.apache.org/activemq/browse/AMQ-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Stirling Chow updated AMQ-2552:
-------------------------------

    Description: 
Symptom
========
If a queue is loaded with 400+ messages that never expire, and then a message that does expire
is added to the end of the queue, the added message will not be processed by the Broker's
expiration logic until some of the 400+ unexpired messages are consumed.  A unit test is provided
to demonstrate this issue.

These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.

Motivation
========
Our application uses message expiration to trigger actions when messages are left in a queue
for too long a period.  In many cases, there are no consumers on the queue containing the
messages that might expire.  Prior versions of AMQ only expired messages when an attempt was
made to dispatch them to a consumer, but AMQ-1112 introduced the notion of periodic, automatic
expiration of pending messages (see the expireMessagesPeriod policy).  With this periodic
expiration, our expectation is that expired messages will be detected and processed in a timely
fashion, regardless of the presence of consumers on the queue.

Cause
======
AMQ-2481 removed forced paging from the periodically-called expireMessages() method:

>>
        doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
<<
        doBrowse(browsedMessages, this.getMaxExpirePageSize());

The purpose of the change was to limit the number of messages that would be paged into memory
from the message store.  The side effect is that only the first 400 messages in the queue
are ever inspected by expireMessages().  If these 400 messages do not contain the messages
that are expiring and there are no active consumers to move the expiring messages forward,
then expiration will never occur.  With AMQ 5.3, successive calls to expireMessages() would
eventually pull all messages into the page so they could be checked for expiration.

It should also be noted that the test cases can be modified so there is a consumer (with a
selector that doesn't choose any of the messages), and they still fail showing that the issue
is not purely the result of having *no* consumers.

Resolution
=========
I don't think the browse idiom works very well for the automated expiration task.  It was
my initial assumption (and expectation) that the expireMessagesPeriod task would process *all*
expired messages in the queue *each time* the task fired.  A related issue to limiting the
number of messages that are examined during each expiration task is that expiring messages
at the end of the queue will not be handled in a timely fashion since they must wait for multiple
firings of the expiration task (this is on 5.3 code; 5.3.1 code will never get to those messages).

If the desire/need is to scan all messages to determine which have expired, the existing paging
strategy is limited because it would require all messages to be brought from store into memory,
thus defeating the whole point of paging.  An alternative might be to have a sliding page
window that would allow batches of messages to be brought in while clearing out the previous
batch.

Another solution might be to have the expiration task page in directly from the store, but
this is problematic since non-persistent messages would still have to come in from the pending
message cursor.  In addition, a complete scan of the store might be non-performant for large
queues.

I think the most satisfactory solution for minimizing unnecessary processing and providing
timely expiration of messages is to implement a proper heap that indexes messages with expiration
values and allows immediate access to *only* the messages that have expired.  The references
to these expired messages could then be used to remove the messages directly from the cursors
and the store rather than having to iterate over these structures.


  was:
Symptom
========
If a queue is loaded with 400+ messages that never expire, and then a message that does expire
is added to the end of the queue, the added message will not be processed by the Broker's
expiration logic until some of the 400+ unexpired messages are consumed.  A unit test is provided
to demonstrate this issue.

These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.

Motivation
========
Our application uses message expiration to trigger actions when messages are left in a queue
for too long a period.  In many cases, there are no consumers on the queue containing the
messages that might expire.  Prior versions of AMQ only expired messages when an attempt was
made to dispatch them to a consumer, but AMQ-1112 introduced the notion of periodic, automatic
expiration of pending messages (see the expireMessagesPeriod policy).  With this periodic
expiration, our expectation is that expired messages will be detected and processed in a timely
fashion, regardless of the presence of consumers on the queue.

Cause
======
AMQ-2481 removed forced paging from the periodically-called expireMessages() method:

>>
        doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
<<
        doBrowse(browsedMessages, this.getMaxExpirePageSize());

The purpose of the change was to limit the number of messages that would be paged into memory
from the message store.  The side effect is that only the first 400 messages in the queue
are ever inspected by expireMessages().  If these 400 messages do not contain the messages
that are expiring and there are no active consumers to move the expiring messages forward,
then expiration will never occur.  With AMQ 5.3, successive calls to expireMessages() would
eventually pull all messages into the page so they could be checked for expiration.

It should also be noted that the test cases can be modified so there is a consumer (with a
selector that doesn't choose any of the messages), and they still fail showing that the issue
is not purely the result of having *no* consumers.

Resolution
=========
I don't think the browse idiom works very well for the automated expiration task.  It was
my initial assumption (and expectation) that the expireMessagesPeriod task would process *all*
expired messages in the queue *each time* the task fired.  A related issue to limiting the
number of messages that are examined during each expiration task is that expiring messages
at the end of the queue will not be handled in a timely fashion since they must wait for multiple
firings of the expiration task (this is on 5.3 code; 5.3.1 code will never get to those messages).

If the desire/need is to scan all messages to determine which have expired, the existing paging
strategy is limited because it would require all messages to be brought from store into memory,
thus defeating the whole point of paging.  An alternative might be to have a sliding page
window that would allow batches of messages to be brought in while clearing out the previous
batch.

Another solution might be to have the expiration task page in directly from the store, but
this is problematic since non-persistent messages would still have to come in from the pending
message cursor.  In addition, a complete scan of the store might be non-performant for large
queues.

I think the most satisfactory solution for minimizing unecessary processing and providing
timely expiration of messages is to implement a proper heap that indexes messages with expiration
values and allows immediate access to *only* the messages that have expired.  The references
to these expired messages could then be used to remove the messages from the cursors and the
store.



> Automatic message expiration may not occur on a queue with no (or slow) consumers.
> ----------------------------------------------------------------------------------
>
>                 Key: AMQ-2552
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2552
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.3.1
>         Environment: ActiveMQ 5.3.1 Snapshot r893661
>            Reporter: Stirling Chow
>         Attachments: AMQTest.java
>
>
> Symptom
> ========
> If a queue is loaded with 400+ messages that never expire, and then a message that does
expire is added to the end of the queue, the added message will not be processed by the Broker's
expiration logic until some of the 400+ unexpired messages are consumed.  A unit test is provided
to demonstrate this issue.
> These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.
> Motivation
> ========
> Our application uses message expiration to trigger actions when messages are left in
a queue for too long a period.  In many cases, there are no consumers on the queue containing
the messages that might expire.  Prior versions of AMQ only expired messages when an attempt
was made to dispatch them to a consumer, but AMQ-1112 introduced the notion of periodic, automatic
expiration of pending messages (see the expireMessagesPeriod policy).  With this periodic
expiration, our expectation is that expired messages will be detected and processed in a timely
fashion, regardless of the presence of consumers on the queue.
> Cause
> ======
> AMQ-2481 removed forced paging from the periodically-called expireMessages() method:
> >>
>         doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
> <<
>         doBrowse(browsedMessages, this.getMaxExpirePageSize());
> The purpose of the change was to limit the number of messages that would be paged into
memory from the message store.  The side effect is that only the first 400 messages in the
queue are ever inspected by expireMessages().  If these 400 messages do not contain the messages
that are expiring and there are no active consumers to move the expiring messages forward,
then expiration will never occur.  With AMQ 5.3, successive calls to expireMessages() would
eventually pull all messages into the page so they could be checked for expiration.
> It should also be noted that the test cases can be modified so there is a consumer (with
a selector that doesn't choose any of the messages), and they still fail showing that the
issue is not purely the result of having *no* consumers.
> Resolution
> =========
> I don't think the browse idiom works very well for the automated expiration task.  It
was my initial assumption (and expectation) that the expireMessagesPeriod task would process
*all* expired messages in the queue *each time* the task fired.  A related issue to limiting
the number of messages that are examined during each expiration task is that expiring messages
at the end of the queue will not be handled in a timely fashion since they must wait for multiple
firings of the expiration task (this is on 5.3 code; 5.3.1 code will never get to those messages).
> If the desire/need is to scan all messages to determine which have expired, the existing
paging strategy is limited because it would require all messages to be brought from store
into memory, thus defeating the whole point of paging.  An alternative might be to have a
sliding page window that would allow batches of messages to be brought in while clearing out
the previous batch.
> Another solution might be to have the expiration task page in directly from the store,
but this is problematic since non-persistent messages would still have to come in from the
pending message cursor.  In addition, a complete scan of the store might be non-performant
for large queues.
> I think the most satisfactory solution for minimizing unnecessary processing and providing
timely expiration of messages is to implement a proper heap that indexes messages with expiration
values and allows immediate access to *only* the messages that have expired.  The references
to these expired messages could then be used to remove the messages directly from the cursors
and the store rather than having to iterate over these structures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message