activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "SunGard Global Services Germany (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMQ-4556) Store based cursor "forgets" messages
Date Sun, 26 May 2013 14:11:19 GMT

     [ https://issues.apache.org/jira/browse/AMQ-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

SunGard Global Services Germany updated AMQ-4556:
-------------------------------------------------

    Description: 
This issue seems to relate to AMQ-2009 and the referenced articles there.

h3. The issue:
After a give period of time, the broker seems to "forget" about received messages on a queue.
Receiving of new messages and delivering them is unaffected. Just some are lost in the store.
Checking the DB tables, the missing messages do exists there.
ActiveMQ statistics counters are also aware of them and count those messages as "pending"
in the ActiveMQ console (first colummn). But open the details of a queue in ActiveMQ console
do not show them at all.

h3. Analysis so far
We tried several settings for the queues which had no impact on the issue:
* Activating/Deactivating useCache
* Setting different prefetch values from 0, 1, 100 (at least 1 seems to relax the issue a
little bit)
* KahaDB and JDBC persistent (Oracle and MSSQL(jTDS)) are affected

Also some characteristics about affected/not affected queues might help to analyse this further:
* We have one queue with just one producer and many consumers whereas the message is quite
small (just headers): No problem on that queue even after thousend of messages
* Queues with multiple producers and consumers and payloaded text-messages seem to be unaffected
- Maybe the JDBC persistent store trottels the processing enough to "solve" the issue
* Queues with multiple producers and consumers and small messages (just headers) seem to "enable"
the issue. Even with few messages the issue appears

h3. "Recovering" lost messages
Shutdown the (master) broker and restart it. With failover transport this happens transparent
for the clients.
On master startup, all messages from the store (DB) are scanned and "rediscovered"

h3. Workaround
Use another cursor - VM seems to be fine. Additional settings might be required to handle
shotcommings of the VM cursor (max memory, queue memory etc.)
{code:xml}
                <policyEntry queue=">" ...>
                  <pendingQueuePolicy>
                    <vmQueueCursor/>
                  </pendingQueuePolicy>
                </policyEntry>
{code}

h3. Test code
I was not able to create a self-contained unit test.
But the attached code can be used to reproduce the issue quite reliable.
It sends 3000 messages. WIth the following settings it will fail/will work correctly:
|| Message Size || ActiveMQ cursor || Result ||
| 1 MB (leave line 20 as it is) | vmQueueCursor | (/) - All messages delivered|
| 1 MB (leave line 20 as it is) | store based Cursor| (/) - All messages delivered|
| remove comment from line 20 | vmQueueCursor| (/) - All messages delivered|
| remove comment from line 20 | store based Cursor| (x) - Some (~0,5% messages lost)|

For completness also the used activemq-broker.xml was attached - This is the default one with
the JDBC persistent store added.

  was:
This issue seems to relate to AMQ-2009 and the referenced articles there.

h3. The issue:
After a give period of time, the broker seems to "forget" about received messages on a queue.
Receiving of new messages and delivering them is unaffected. Just some are lost in the store.
Checking the DB tables, the missing messages do exists there.
ActiveMQ statistics counters are also aware of them and count those messages as "pending"
in the ActiveMQ console (first colummn). But open the details of a queue in ActiveMQ console
do not show them at all.

h3. Analysis so far
We tried several settings for the queues which had no impact on the issue:
* Activating/Deactivating useCache
* Setting different prefetch values from 0, 1, 100 (at least 1 seems to relax the issue a
little bit)
* KahaDB and JDBC persistent (Oracle and MSSQL(jTDS)) are affected

Also some characteristics about affected/not affected queues might help to analyse this further:
* We have one queue with just one producer and many consumers whereas the message is quite
small (just headers): No problem on that queue even after thousend of messages
* Queues with multiple producers and consumers and payloaded text-messages seem to be unaffected
- Maybe the JDBC persistent store trottels the processing enough to "solve" the issue
* Queues with multiple producers and consumers and small messages (just headers) seem to "enable"
the issue. Even with few messages the issue appears

h3. "Recovering" lost messages
Shutdown the (master) broker and restart it. With failover transport this happens transparent
for the clients.
On master startup, all messages from the store (DB) are scanned and "rediscovered"

h3. Workaround
Use another cursor - VM seems to be fine. Additional settings might be required to handle
shotcommings of the VM cursor (max memory, queue memory etc.)
{code:xml}
                <policyEntry queue=">" ...>
                  <pendingQueuePolicy>
                    <vmQueueCursor/>
                  </pendingQueuePolicy>
                </policyEntry>
{code}

h3. Test code
I was not able to create a self-contained unit test.
But the attached code can be used to reproduce the issue quite reliable.
It sends 3000 messages. WIth the following settings it will fail/will work correctly:
|| Message Size || ActiveMQ cursor || Result ||
| 1 MB (leave line 20 as it is) | vmQueueCursor | (/) |
| 1 MB (leave line 20 as it is) | store based Cursor| (/) |
| remove comment from line 20 | vmQueueCursor| (/) |
| remove comment from line 20 | store based Cursor| (x) |

For completness also the used activemq-broker.xml was attached - This is the default one with
the JDBC persistent store added.

    
> Store based cursor "forgets" messages
> -------------------------------------
>
>                 Key: AMQ-4556
>                 URL: https://issues.apache.org/jira/browse/AMQ-4556
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.7.0, 5.8.0
>         Environment: Durable queues with JDBC persistent store in a master/slave configuration
with NIO transport
>            Reporter: SunGard Global Services Germany
>            Priority: Critical
>         Attachments: ActiveMQClient.java, activemq.xml
>
>
> This issue seems to relate to AMQ-2009 and the referenced articles there.
> h3. The issue:
> After a give period of time, the broker seems to "forget" about received messages on
a queue.
> Receiving of new messages and delivering them is unaffected. Just some are lost in the
store.
> Checking the DB tables, the missing messages do exists there.
> ActiveMQ statistics counters are also aware of them and count those messages as "pending"
in the ActiveMQ console (first colummn). But open the details of a queue in ActiveMQ console
do not show them at all.
> h3. Analysis so far
> We tried several settings for the queues which had no impact on the issue:
> * Activating/Deactivating useCache
> * Setting different prefetch values from 0, 1, 100 (at least 1 seems to relax the issue
a little bit)
> * KahaDB and JDBC persistent (Oracle and MSSQL(jTDS)) are affected
> Also some characteristics about affected/not affected queues might help to analyse this
further:
> * We have one queue with just one producer and many consumers whereas the message is
quite small (just headers): No problem on that queue even after thousend of messages
> * Queues with multiple producers and consumers and payloaded text-messages seem to be
unaffected - Maybe the JDBC persistent store trottels the processing enough to "solve" the
issue
> * Queues with multiple producers and consumers and small messages (just headers) seem
to "enable" the issue. Even with few messages the issue appears
> h3. "Recovering" lost messages
> Shutdown the (master) broker and restart it. With failover transport this happens transparent
for the clients.
> On master startup, all messages from the store (DB) are scanned and "rediscovered"
> h3. Workaround
> Use another cursor - VM seems to be fine. Additional settings might be required to handle
shotcommings of the VM cursor (max memory, queue memory etc.)
> {code:xml}
>                 <policyEntry queue=">" ...>
>                   <pendingQueuePolicy>
>                     <vmQueueCursor/>
>                   </pendingQueuePolicy>
>                 </policyEntry>
> {code}
> h3. Test code
> I was not able to create a self-contained unit test.
> But the attached code can be used to reproduce the issue quite reliable.
> It sends 3000 messages. WIth the following settings it will fail/will work correctly:
> || Message Size || ActiveMQ cursor || Result ||
> | 1 MB (leave line 20 as it is) | vmQueueCursor | (/) - All messages delivered|
> | 1 MB (leave line 20 as it is) | store based Cursor| (/) - All messages delivered|
> | remove comment from line 20 | vmQueueCursor| (/) - All messages delivered|
> | remove comment from line 20 | store based Cursor| (x) - Some (~0,5% messages lost)|
> For completness also the used activemq-broker.xml was attached - This is the default
one with the JDBC persistent store added.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message