activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Vallens (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMQ-3866) kahadb log files not getting cleaned up
Date Wed, 30 May 2012 00:17:23 GMT
Alexander Vallens created AMQ-3866:
--------------------------------------

             Summary: kahadb log files not getting cleaned up
                 Key: AMQ-3866
                 URL: https://issues.apache.org/jira/browse/AMQ-3866
             Project: ActiveMQ
          Issue Type: Bug
          Components: Message Store
    Affects Versions: 5.6.0, 5.5.1
            Reporter: Alexander Vallens
            Priority: Critical


A few weeks ago we experienced an event where several hundred thousand messages were posted
to a queue and slowly consumed over several days until accidentally purged. Ever since then
our kahadb has been growing on the order of 12-20 bytes/10 seconds, corresponding to additional
entries being added to the kahadb log files that are not being cleaned up (this is even the
case in an isolated test environment with no subscribers/consumers/activity). Having searched
and found AMQ-3131, AMQ-3736, AMQ-3424, we attempted an upgrade on a clone of the affected
system from 5.5.1 to 5.6.0, with no luck. Along the way here are the steps we took:

1)	I cloned our ActiveMQ virtual machine “hot” (i.e. without shutdown) and was able to
get an isolated (different hostname, IP, VLAN), but functioning, “copy” of the production
ActiveMQ instance
2)	startup is slow, given the sheer size of the kahadb, as shown in the startup logging:
2012-05-29 19:59:11,780 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 20:05:28,352 | WARN  | Duplicate message add attempt rejected. Destination: Consumer.SchedulerEvents.VirtualTopic.scheduler/events,
Message id: ID:<FQDN>-55381-1337386136713-0:18:1:1:1 | org.apache.a
ctivemq.store.kahadb.MessageDatabase | main
2012-05-29 20:05:28,494 | WARN  | Duplicate message add attempt rejected. Destination: Consumer.SchedulerEvents.VirtualTopic.scheduler/events,
Message id: ID:<FQDN>-55381-1337386136713-0:25:1:1:1 | org.apache.a
ctivemq.store.kahadb.MessageDatabase | main
2012-05-29 20:05:28,528 | WARN  | Duplicate message add attempt rejected. Destination: Consumer.SchedulerEvents.VirtualTopic.scheduler/events,
Message id: ID:<FQDN>-55381-1337386136713-0:32:1:1:1 | org.apache.a
ctivemq.store.kahadb.MessageDatabase | main
2012-05-29 20:05:34,513 | INFO  | Recovery replayed 2325747 operations from the journal in
382.795 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
3)	I was able to eliminate the 3 “duplicate message” warnings by purging the “Consumer.SchedulerEvents.VirtualTopic.scheduler/events”
queue (I suspect there were unconsumed messages, perhaps partially uploaded, in that queue
at time of clone)
4)	after the above steps, I was able to observe that kahadb continued to grow at the same
rate as the actual production ActiveMQ instancw, ~12-20 bytes every 10s
5)	during the first normal shutdown of activemq, I observed the following errors:
java.lang.IllegalStateException: PageFile is not loaded
        at org.apache.kahadb.page.PageFile.assertLoaded(PageFile.java:715)
        at org.apache.kahadb.page.PageFile.tx(PageFile.java:239)
        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.resetBatching(KahaDBStore.java:510)
        at org.apache.activemq.store.ProxyMessageStore.resetBatching(ProxyMessageStore.java:93)
        at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.resetBatch(QueueStorePrefetch.java:85)
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.stop(AbstractStoreCursor.java:67)
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.stop(StoreQueueCursor.java:84)
        at org.apache.activemq.broker.region.Queue.stop(Queue.java:872)
        at org.apache.activemq.broker.region.AbstractRegion.stop(AbstractRegion.java:110)
        at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
        at org.apache.activemq.broker.region.RegionBroker.doStop(RegionBroker.java:729)
        at org.apache.activemq.broker.jmx.ManagedRegionBroker.doStop(ManagedRegionBroker.java:113)
        at org.apache.activemq.broker.region.RegionBroker.stop(RegionBroker.java:213)
        at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
        at org.apache.activemq.broker.scheduler.SchedulerBroker.stop(SchedulerBroker.java:104)
        at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
        at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
        at org.apache.activemq.broker.TransactionBroker.stop(TransactionBroker.java:114)
        at org.apache.activemq.broker.BrokerService$3.stop(BrokerService.java:1795)
        at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
        at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:587)
        at org.apache.activemq.console.command.StartCommand$1.run(StartCommand.java:135)
2012-05-29 20:22:09,112 | ERROR | Failed to reset batching | org.apache.activemq.store.kahadb.KahaDBStore
| Thread-45
6)	subsequent shutdowns, however, operated normally without error
7)	even after various attempts at shutdown and start back up, the aforementioned journal recovery
was time consuming:
2012-05-29 20:25:06,780 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 20:31:39,821 | INFO  | Recovery replayed 2325954 operations from the journal in
393.074 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
8)	and the kahadb growth issue persisted
9)	I went ahead and successfully upgraded to ActiveMQ v5.6.0 (verified queues & messages
in scheduled queue survived upgrade)
10)	first attempt to start activemq after 5.6.0 upgrade included following errors:
2012-05-29 20:55:36,825 | ERROR | Failed to start ActiveMQ JMS Message Broker (localhost,
null). Reason: java.lang.NegativeArraySizeException | org.apache.activemq.broker.BrokerService
| main
java.lang.NegativeArraySizeException
        at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:188)
        at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
        at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
        at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
        at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
        at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
        at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
        at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller$1.execute(MessageDatabase.java:1654)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
        at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1642)
        at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1627)
        at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:201)
        at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
        at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
        at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
        at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
        at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
        at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
        at org.apache.activemq.store.kahadb.KahaDBStore$6.execute(KahaDBStore.java:943)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
        at org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:941)
        at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:122)

        at org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)
        at org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2369)
        at org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2198)
        at org.apache.activemq.broker.BrokerService.start(BrokerService.java:527)
        at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)

        at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:108)
        at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:108)
        at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:72)
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
        at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:257)
        at org.apache.activemq.console.Main.main(Main.java:111)
2012-05-29 20:55:36,873 | INFO  | ActiveMQ Message Broker (localhost, null) is shutting down
| org.apache.activemq.broker.BrokerService | main
2012-05-29 20:55:36,873 | INFO  | Connector openwire Stopped | org.apache.activemq.broker.TransportConnector
| main
2012-05-29 20:55:36,873 | INFO  | Connector stomp Stopped | org.apache.activemq.broker.TransportConnector
| main
2012-05-29 20:55:36,897 | INFO  | PListStore:[/opt/activemq/data/localhost/tmp_storage ] stopped
| org.apache.activemq.store.kahadb.plist.PListStore | main
2012-05-29 20:55:36,897 | INFO  | Stopping async queue tasks | org.apache.activemq.store.kahadb.KahaDBStore
| main
2012-05-29 20:55:36,897 | INFO  | Stopping async topic tasks | org.apache.activemq.store.kahadb.KahaDBStore
| main 
2012-05-29 20:55:36,897 | INFO  | Stopped KahaDB | org.apache.activemq.store.kahadb.KahaDBStore
| main
2012-05-29 20:55:37,297 | INFO  | ActiveMQ JMS Message Broker (localhost, null) stopped |
org.apache.activemq.broker.BrokerService | main
2012-05-29 20:55:37,300 | ERROR | Failed to load: class path resource [activemq.xml], reason:
Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in
class path resource [activemq.xml]: Invoc
ation of init method failed; nested exception is java.lang.NegativeArraySizeException | org.apache.activemq.xbean.XBeanBrokerFactory
| main
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0'
defined in class path resource [activemq.xml]: Invocation of init method failed; nested e
xception is java.lang.NegativeArraySizeException
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
        at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:108)
        at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:108)
        at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:72)
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54) 
        at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:257)
        at org.apache.activemq.console.Main.main(Main.java:111)
Caused by: java.lang.NegativeArraySizeException
        at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:188)
        at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
        at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
        at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
        at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
        at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
        at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
        at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller$1.execute(MessageDatabase.java:1654)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
        at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1642)

        at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1627)
        at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:201)
        at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
        at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
        at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
        at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
        at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
        at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
        at org.apache.activemq.store.kahadb.KahaDBStore$6.execute(KahaDBStore.java:943)
        at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
        at org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:941)
        at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:122)
        at org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)
        at org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2369)
        at org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2198)
        at org.apache.activemq.broker.BrokerService.start(BrokerService.java:527)
        at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
        ... 28 more
11)	after the above errors, ActiveMQ was dead; another attempt to start worked, yielding the
following (also logged in previous attempt):
2012-05-29 21:58:01,317 | INFO  | KahaDB is version 4 | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 21:58:01,396 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 21:58:27,601 | INFO  | @2207:10939359, 100000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 21:58:42,788 | INFO  | @2222:10328575, 200000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 21:58:59,234 | INFO  | @2242:14360209, 300000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 21:59:13,628 | INFO  | @2254:1365404, 400000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 21:59:27,466 | INFO  | @2264:30387363, 500000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 21:59:43,739 | INFO  | @2281:28756842, 600000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:00:13,389 | INFO  | @2320:26490711, 700000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:00:47,350 | INFO  | @2371:21837918, 800000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:01:17,752 | INFO  | @2412:10702462, 900000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:01:51,288 | INFO  | @2465:32070937, 1000000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:02:25,203 | INFO  | @2519:21797589, 1100000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:02:55,188 | INFO  | @2564:13289299, 1200000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:03:05,786 | INFO  | @2572:23712519, 1300000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:03:15,375 | INFO  | @2578:22618226, 1400000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:03:27,246 | INFO  | @2585:31399538, 1500000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:03:38,213 | INFO  | @2593:7102179, 1600000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:03:48,285 | INFO  | @2600:11040885, 1700000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:03:55,053 | INFO  | @2602:28573821, 1800000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:04:01,331 | INFO  | @2603:32850602, 1900000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:04:06,689 | INFO  | @2605:3582539, 2000000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:04:12,435 | INFO  | @2606:7920235, 2100000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:04:17,500 | INFO  | @2607:12248010, 2200000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main 
2012-05-29 22:04:25,521 | INFO  | @2617:28414362, 2300000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2012-05-29 22:04:28,459 | INFO  | Recovery replayed 2326036 operations from the journal in
387.094 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
12)	unfortunately, I confirmed that even after upgrade, kahadb continues to grow at about
the same rate as before
13) I then applied the logging parameters as described in http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html.
. With debugging spun up we see A LOT (I can’t get a count because log files rotate too
fast, but in the 6 logs that remain I get 31,978 entries) of the following:
2012-05-29 23:18:41,925 | DEBUG | message not found in sequence id index: ID:<FQDN>-59596-1337475432262-0:35:1:1:1202
| org.apache.activemq.store.kahadb.MessageDatabase | main

Google and a search on issues.apache.org reveal nothing on this.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message