activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Vallens (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-3866) kahadb log files not getting cleaned up
Date Thu, 07 Jun 2012 16:25:24 GMT

    [ https://issues.apache.org/jira/browse/AMQ-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291108#comment-13291108
] 

Alexander Vallens commented on AMQ-3866:
----------------------------------------

Has anyone seen this bug? If it's invalid, please let us know.
                
> kahadb log files not getting cleaned up
> ---------------------------------------
>
>                 Key: AMQ-3866
>                 URL: https://issues.apache.org/jira/browse/AMQ-3866
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.5.1, 5.6.0
>            Reporter: Alexander Vallens
>            Priority: Critical
>
> A few weeks ago we experienced an event where several hundred thousand messages were
posted to a queue and slowly consumed over several days until accidentally purged. Ever since
then our kahadb has been growing on the order of 12-20 bytes/10 seconds, corresponding to
additional entries being added to the kahadb log files that are not being cleaned up (this
is even the case in an isolated test environment with no subscribers/consumers/activity).
Having searched and found AMQ-3131, AMQ-3736, AMQ-3424, we attempted an upgrade on a clone
of the affected system from 5.5.1 to 5.6.0, with no luck. Along the way here are the steps
we took:
> 1)	I cloned our ActiveMQ virtual machine “hot” (i.e. without shutdown) and was able
to get an isolated (different hostname, IP, VLAN), but functioning, “copy” of the production
ActiveMQ instance
> 2)	startup is slow, given the sheer size of the kahadb, as shown in the startup logging:
> 2012-05-29 19:59:11,780 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 20:05:28,352 | WARN  | Duplicate message add attempt rejected. Destination:
Consumer.SchedulerEvents.VirtualTopic.scheduler/events, Message id: ID:<FQDN>-55381-1337386136713-0:18:1:1:1
| org.apache.a
> ctivemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:28,494 | WARN  | Duplicate message add attempt rejected. Destination:
Consumer.SchedulerEvents.VirtualTopic.scheduler/events, Message id: ID:<FQDN>-55381-1337386136713-0:25:1:1:1
| org.apache.a
> ctivemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:28,528 | WARN  | Duplicate message add attempt rejected. Destination:
Consumer.SchedulerEvents.VirtualTopic.scheduler/events, Message id: ID:<FQDN>-55381-1337386136713-0:32:1:1:1
| org.apache.a
> ctivemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:34,513 | INFO  | Recovery replayed 2325747 operations from the journal
in 382.795 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 3)	I was able to eliminate the 3 “duplicate message” warnings by purging the “Consumer.SchedulerEvents.VirtualTopic.scheduler/events”
queue (I suspect there were unconsumed messages, perhaps partially uploaded, in that queue
at time of clone)
> 4)	after the above steps, I was able to observe that kahadb continued to grow at the
same rate as the actual production ActiveMQ instancw, ~12-20 bytes every 10s
> 5)	during the first normal shutdown of activemq, I observed the following errors:
> java.lang.IllegalStateException: PageFile is not loaded
>         at org.apache.kahadb.page.PageFile.assertLoaded(PageFile.java:715)
>         at org.apache.kahadb.page.PageFile.tx(PageFile.java:239)
>         at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.resetBatching(KahaDBStore.java:510)
>         at org.apache.activemq.store.ProxyMessageStore.resetBatching(ProxyMessageStore.java:93)
>         at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.resetBatch(QueueStorePrefetch.java:85)
>         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.stop(AbstractStoreCursor.java:67)
>         at org.apache.activemq.broker.region.cursors.StoreQueueCursor.stop(StoreQueueCursor.java:84)
>         at org.apache.activemq.broker.region.Queue.stop(Queue.java:872)
>         at org.apache.activemq.broker.region.AbstractRegion.stop(AbstractRegion.java:110)
>         at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
>         at org.apache.activemq.broker.region.RegionBroker.doStop(RegionBroker.java:729)
>         at org.apache.activemq.broker.jmx.ManagedRegionBroker.doStop(ManagedRegionBroker.java:113)
>         at org.apache.activemq.broker.region.RegionBroker.stop(RegionBroker.java:213)
>         at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
>         at org.apache.activemq.broker.scheduler.SchedulerBroker.stop(SchedulerBroker.java:104)
>         at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
>         at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
>         at org.apache.activemq.broker.TransactionBroker.stop(TransactionBroker.java:114)
>         at org.apache.activemq.broker.BrokerService$3.stop(BrokerService.java:1795)
>         at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
>         at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:587)
>         at org.apache.activemq.console.command.StartCommand$1.run(StartCommand.java:135)
> 2012-05-29 20:22:09,112 | ERROR | Failed to reset batching | org.apache.activemq.store.kahadb.KahaDBStore
| Thread-45
> 6)	subsequent shutdowns, however, operated normally without error
> 7)	even after various attempts at shutdown and start back up, the aforementioned journal
recovery was time consuming:
> 2012-05-29 20:25:06,780 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 20:31:39,821 | INFO  | Recovery replayed 2325954 operations from the journal
in 393.074 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 8)	and the kahadb growth issue persisted
> 9)	I went ahead and successfully upgraded to ActiveMQ v5.6.0 (verified queues & messages
in scheduled queue survived upgrade)
> 10)	first attempt to start activemq after 5.6.0 upgrade included following errors:
> 2012-05-29 20:55:36,825 | ERROR | Failed to start ActiveMQ JMS Message Broker (localhost,
null). Reason: java.lang.NegativeArraySizeException | org.apache.activemq.broker.BrokerService
| main
> java.lang.NegativeArraySizeException
>         at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:188)
>         at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
>         at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
>         at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
>         at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
>         at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
>         at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
>         at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller$1.execute(MessageDatabase.java:1654)
>         at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
>         at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1642)
>         at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1627)
>         at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:201)
>         at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
>         at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
>         at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
>         at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
>         at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
>         at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
>         at org.apache.activemq.store.kahadb.KahaDBStore$6.execute(KahaDBStore.java:943)
>         at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
>         at org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:941)
>         at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:122)

>         at org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)
>         at org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2369)
>         at org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2198)
>         at org.apache.activemq.broker.BrokerService.start(BrokerService.java:527)
>         at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
>         at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
>         at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
>         at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
>         at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
>         at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
>         at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
>         at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
>         at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
>         at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)

>         at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:108)
>         at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:108)
>         at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:72)
>         at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
>         at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
>         at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
>         at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
>         at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
>         at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
>         at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
>         at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.activemq.console.Main.runTaskClass(Main.java:257)
>         at org.apache.activemq.console.Main.main(Main.java:111)
> 2012-05-29 20:55:36,873 | INFO  | ActiveMQ Message Broker (localhost, null) is shutting
down | org.apache.activemq.broker.BrokerService | main
> 2012-05-29 20:55:36,873 | INFO  | Connector openwire Stopped | org.apache.activemq.broker.TransportConnector
| main
> 2012-05-29 20:55:36,873 | INFO  | Connector stomp Stopped | org.apache.activemq.broker.TransportConnector
| main
> 2012-05-29 20:55:36,897 | INFO  | PListStore:[/opt/activemq/data/localhost/tmp_storage
] stopped | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2012-05-29 20:55:36,897 | INFO  | Stopping async queue tasks | org.apache.activemq.store.kahadb.KahaDBStore
| main
> 2012-05-29 20:55:36,897 | INFO  | Stopping async topic tasks | org.apache.activemq.store.kahadb.KahaDBStore
| main 
> 2012-05-29 20:55:36,897 | INFO  | Stopped KahaDB | org.apache.activemq.store.kahadb.KahaDBStore
| main
> 2012-05-29 20:55:37,297 | INFO  | ActiveMQ JMS Message Broker (localhost, null) stopped
| org.apache.activemq.broker.BrokerService | main
> 2012-05-29 20:55:37,300 | ERROR | Failed to load: class path resource [activemq.xml],
reason: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined
in class path resource [activemq.xml]: Invoc
> ation of init method failed; nested exception is java.lang.NegativeArraySizeException
| org.apache.activemq.xbean.XBeanBrokerFactory | main
> org.springframework.beans.factory.BeanCreationException: Error creating bean with name
'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]:
Invocation of init method failed; nested e
> xception is java.lang.NegativeArraySizeException
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
>         at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
>         at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
>         at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
>         at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
>         at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
>         at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
>         at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
>         at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
>         at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
>         at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:108)
>         at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:108)
>         at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:72)
>         at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
>         at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)

>         at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
>         at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
>         at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
>         at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
>         at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
>         at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.activemq.console.Main.runTaskClass(Main.java:257)
>         at org.apache.activemq.console.Main.main(Main.java:111)
> Caused by: java.lang.NegativeArraySizeException
>         at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:188)
>         at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
>         at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
>         at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
>         at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
>         at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
>         at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
>         at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller$1.execute(MessageDatabase.java:1654)
>         at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
>         at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1642)

>         at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1627)
>         at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:201)
>         at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
>         at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
>         at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
>         at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
>         at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
>         at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
>         at org.apache.activemq.store.kahadb.KahaDBStore$6.execute(KahaDBStore.java:943)
>         at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
>         at org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:941)
>         at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:122)
>         at org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)
>         at org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2369)
>         at org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2198)
>         at org.apache.activemq.broker.BrokerService.start(BrokerService.java:527)
>         at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544)
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485)

>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
>         ... 28 more
> 11)	after the above errors, ActiveMQ was dead; another attempt to start worked, yielding
the following (also logged in previous attempt):
> 2012-05-29 21:58:01,317 | INFO  | KahaDB is version 4 | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 21:58:01,396 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 21:58:27,601 | INFO  | @2207:10939359, 100000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 21:58:42,788 | INFO  | @2222:10328575, 200000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 21:58:59,234 | INFO  | @2242:14360209, 300000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 21:59:13,628 | INFO  | @2254:1365404, 400000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 21:59:27,466 | INFO  | @2264:30387363, 500000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 21:59:43,739 | INFO  | @2281:28756842, 600000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:00:13,389 | INFO  | @2320:26490711, 700000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:00:47,350 | INFO  | @2371:21837918, 800000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:01:17,752 | INFO  | @2412:10702462, 900000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:01:51,288 | INFO  | @2465:32070937, 1000000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:02:25,203 | INFO  | @2519:21797589, 1100000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:02:55,188 | INFO  | @2564:13289299, 1200000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:03:05,786 | INFO  | @2572:23712519, 1300000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:03:15,375 | INFO  | @2578:22618226, 1400000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:03:27,246 | INFO  | @2585:31399538, 1500000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:03:38,213 | INFO  | @2593:7102179, 1600000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:03:48,285 | INFO  | @2600:11040885, 1700000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:03:55,053 | INFO  | @2602:28573821, 1800000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:04:01,331 | INFO  | @2603:32850602, 1900000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:04:06,689 | INFO  | @2605:3582539, 2000000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:04:12,435 | INFO  | @2606:7920235, 2100000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:04:17,500 | INFO  | @2607:12248010, 2200000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main 
> 2012-05-29 22:04:25,521 | INFO  | @2617:28414362, 2300000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase
| main
> 2012-05-29 22:04:28,459 | INFO  | Recovery replayed 2326036 operations from the journal
in 387.094 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 12)	unfortunately, I confirmed that even after upgrade, kahadb continues to grow at about
the same rate as before
> 13) I then applied the logging parameters as described in http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html.
. With debugging spun up we see A LOT (I can’t get a count because log files rotate too
fast, but in the 6 logs that remain I get 31,978 entries) of the following:
> 2012-05-29 23:18:41,925 | DEBUG | message not found in sequence id index: ID:<FQDN>-59596-1337475432262-0:35:1:1:1202
| org.apache.activemq.store.kahadb.MessageDatabase | main
> Google and a search on issues.apache.org reveal nothing on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message