activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Tully (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-1795) in use data files removed from data store under load
Date Tue, 17 Jun 2008 10:43:00 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=43520#action_43520
] 

Gary Tully commented on AMQ-1795:
---------------------------------

Still having problems with this:

I find that the following change helps but is not bulit proof. I think it makes sense to use
the mark as the boundary point for deletion but on a slower machine the MissingDataFileTest
still shows the error stack trace. A multi core machine does not show the problem.
Also, the test case only complets when the memory usage limit is omitted or increased to 1024*1024
which may indicate a limitation of the test case. Decreasing the memory usage caused the problem
to show up earlier, which does not now seem to be the case. So there is some progress.

The short of it is that collection of inUse file Ids is not sufficiently stable for the cleanup
method. I still do not know where the problem lies though. I need another set of eyes.

Index: src/main/java/org/apache/activemq/store/amq/AMQPersistenceAdapter.java
===================================================================
--- src/main/java/org/apache/activemq/store/amq/AMQPersistenceAdapter.java      (revision
668601)
+++ src/main/java/org/apache/activemq/store/amq/AMQPersistenceAdapter.java      (working copy)
@@ -425,6 +425,7 @@
             }
             Integer lastDataFile = asyncDataManager.getCurrentDataFileId();
             inProgress.add(lastDataFile);
+            lastDataFile = asyncDataManager.getMark().getDataFileId();
             inProgress.addAll(referenceStoreAdapter.getReferenceFileIdsInUse());
             Location lastActiveTx = transactionStore.checkpoint();
             if (lastActiveTx != null) {


> in use data files removed from data store under load
> ----------------------------------------------------
>
>                 Key: AMQ-1795
>                 URL: https://issues.apache.org/activemq/browse/AMQ-1795
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.1.0
>         Environment: all
>            Reporter: Gary Tully
>            Assignee: Rob Davies
>             Fix For: 5.2.0
>
>         Attachments: AMQ-1795.test_and_partial_fix
>
>
> Under load, I see the error below.
> the problem is that org.apache.activemq.store.amq.AMQPersistenceAdapter.cleanup() does
not capture all of the in use files.
> I have made some changes that improves the situation but there is still a window, that
seems to be related to spooling messages when a memory limit is reached.
> 2008-06-12 14:55:16,689 [main           ] INFO  BrokerService                  - Using
Persistence Adapter: AMQPersistenceAdapter(activemq-data\localhost)
> 2008-06-12 14:55:16,704 [main           ] INFO  AMQPersistenceAdapter          - AMQStore
starting using directory: activemq-data\localhost
> 2008-06-12 14:55:17,361 [main           ] INFO  AMQPersistenceAdapter          - Journal
deleted: 
> 2008-06-12 14:55:17,361 [main           ] INFO  KahaStore                      - Kaha
Store successfully deleted data directory activemq-data\localhost\kr-store\data
> 2008-06-12 14:55:17,392 [main           ] INFO  KahaStore                      - Kaha
Store successfully deleted data directory activemq-data\localhost\kr-store\state
> 2008-06-12 14:55:17,392 [main           ] INFO  KahaStore                      - Kaha
Store using data directory activemq-data\localhost\kr-store\state
> 2008-06-12 14:55:17,470 [main           ] INFO  AMQPersistenceAdapter          - Active
data files: []
> 2008-06-12 14:55:17,705 [main           ] INFO  BrokerService                  - ActiveMQ
null JMS Message Broker (localhost) is starting
> 2008-06-12 14:55:17,705 [main           ] INFO  BrokerService                  - For
help or more information please see: http://activemq.apache.org/
> 2008-06-12 14:55:17,814 [JMX connector  ] INFO  ManagementContext              - JMX
consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
> 2008-06-12 14:55:17,924 [main           ] INFO  KahaStore                      - Kaha
Store using data directory activemq-data\localhost\kr-store\data
> 2008-06-12 14:55:17,939 [main           ] INFO  TransportServerThreadSupport   - Listening
for connections at: tcp://salthill:61616
> 2008-06-12 14:55:17,939 [main           ] INFO  TransportConnector             - Connector
Default Started
> 2008-06-12 14:55:17,939 [main           ] INFO  BrokerService                  - ActiveMQ
JMS Message Broker (localhost, ID:salthill-2790-1213278917752-0:0) started
> 2008-06-12 14:55:17,939 [main           ] INFO  MissingDataFileTest            - Starting
broker..
> 2008-06-12 14:55:58,516 [age Thread Pool] ERROR AsyncDataManager               - Looking
for key 551 but not found in fileMap: {1014=data-1014 number = 1014 , length = 1960 refCount
= 6, 607=data-607 number = 607 , length = 1266 refCount = 1, 928=data-928 number = 928 , ..
.. 622 , length = 1352 refCount = 2, 328=data-328 number = 328 , length = 1266 refCount =
1, 256=data-256 number = 256 , length = 1264 refCount = 1, 92=data-92 number = 92 , length
= 1264 refCount = 1, 404=data-404 number = 404 , length = 1266 refCount = 1}
> 2008-06-12 14:55:58,563 [age Thread Pool] ERROR AbstractStoreCursor            - Failed
to fill batch
> java.io.IOException: Failed to read to journal for: offset = 0, file = 551, size = -1,
type = 0. Reason: java.io.IOException: Could not locate data file data--551
> 	at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
> 	at org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:635)
> 	at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:521)
> 	at org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:432)
> 	at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
> 	at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:82)
> 	at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:120)
> 	at org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:530)
> 	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:75)
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.onUsageChanged(AbstractStoreCursor.java:157)
> 	at org.apache.activemq.usage.Usage$1.run(Usage.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> 	at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.IOException: Could not locate data file data--551
> 	at org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:303)
> 	at org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:613)
> 	at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:518)
> 	... 12 more
> 2008-06-12 14:55:58,563 [age Thread Pool] ERROR AbstractStoreCursor            - Failed
to fill batch 
> java.lang.RuntimeException: java.io.IOException: Failed to read to journal for: offset
= 0, file = 551, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file
data--551
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:191)
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.onUsageChanged(AbstractStoreCursor.java:157)
> 	at org.apache.activemq.usage.Usage$1.run(Usage.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> 	at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.IOException: Failed to read to journal for: offset = 0, file = 551,
size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data--551
> 	at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
> 	at org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:635)
> 	at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:521)
> 	at org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:432)
> 	at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
> 	at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:82)
> 	at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:120)
> 	at org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:530)
> 	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:75)
> 	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
> 	... 5 more
> Caused by: java.io.IOException: Could not locate data file data--551
> 	at org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:303)
> 	at org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:613)
> 	at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:518)
> 	... 12 more
> 2008-06-12 15:00:18,531 [main           ] INFO  BrokerService                  - ActiveMQ
Message Broker (localhost, ID:salthill-2790-1213278917752-0:0) is shutting down
> 2008-06-12 15:00:20,438 [main           ] INFO  TransportConnector             - Connector
Default Stopped
> 2008-06-12 15:00:20,641 [main           ] INFO  BrokerService                  - ActiveMQ
JMS Message Broker (localhost, ID:salthill-2790-1213278917752-0:0) stopped

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message