activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Klaus Pittig (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AMQ-6011) create a how-to about repair/recover corrupt LevelDB
Date Thu, 15 Oct 2015 18:41:05 GMT

    [ https://issues.apache.org/jira/browse/AMQ-6011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958992#comment-14958992
] 

Klaus Pittig edited comment on AMQ-6011 at 10/15/15 6:40 PM:
-------------------------------------------------------------

I was able to repair this leveldb store with minimum loss succesfully for Windows by calling
{code}
org.fusesource.leveldbjni.JniDBFactory.factory.repair(...);
{code}
on each of the *.index directories (not the storage directory).
To get ActiveMQ up&running again, I had to remove the "lost" directory afterwarda due
to some access restriction ActiveMQ complains about.


Kindly excuse, if this seems a bit curious to you. I just wanted to find the fastest way for
me at this time.



was (Author: jforge):
I was able to repair this leveldb store with minimum loss succesfully for windows by calling
{code}
org.fusesource.leveldbjni.JniDBFactory.factory.repair(...);
{code}
on each of the *.index directories (not the storage directory).
To get ActiveMQ up&running again, I had to remove the "lost" directory afterward due to
some access restriction, ActiveMQ complains about.


Kindly excuse, of this seems a bit curious to you. I just wanted to find the fastest way for
me at this time.


> create a how-to about repair/recover corrupt LevelDB
> ----------------------------------------------------
>
>                 Key: AMQ-6011
>                 URL: https://issues.apache.org/jira/browse/AMQ-6011
>             Project: ActiveMQ
>          Issue Type: Improvement
>          Components: Message Store
>    Affects Versions: 5.11.2
>         Environment: Windows x64 7,8; Oracle Java 8 (u60)
>            Reporter: Klaus Pittig
>
> With a single AMQ 5.11.2 instance, Java8u60 and Windows 7+ we can produce LevelDB corruption
by putting the system to sleep (sometimes by simply closing a notebook).
> *BTW*: The LevelDB auto-recovery on the corrupted LevelDBStore works perfectly on a Linux
or MacOS system, but unfortunately this is not an option in some customer infrastructure.
> Irrespective of whether this bug should have been recovered by the LevelDB store automatically
or not, how can this situation be recovered (more or less manually) _without_ losing the persistent
queue content? The KahaDB recovery docs were enough in most situations, and something near
it with LevelDB could help not to switch back to KahaDB or other Persistence Adapters.
> The above mentioned scenario produces the following log output, how can this be fixed
reliably with a minimum of data loss?
> {code}
> jvm 1    | ERROR: org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource
[activemq.xml]: Invocation of init method failed; nested exception is org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException:
java.lang.RuntimeException: Could not open table 11
> jvm 1    | org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource
[activemq.xml]: Invocation of init method failed; nested exception is org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException:
java.lang.RuntimeException: Could not open table 11
> jvm 1    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1514)
> jvm 1    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
> jvm 1    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
> jvm 1    | 	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
> jvm 1    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
> jvm 1    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
> jvm 1    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
> jvm 1    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:636)
> jvm 1    | 	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:934)
> jvm 1    | 	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
> jvm 1    | 	at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
> jvm 1    | 	at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
> jvm 1    | 	at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)
> jvm 1    | 	at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)
> jvm 1    | 	at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)
> jvm 1    | 	at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
> jvm 1    | 	at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
> jvm 1    | 	at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)
> jvm 1    | 	at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> jvm 1    | 	at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
> jvm 1    | 	at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> jvm 1    | 	at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
> jvm 1    | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> jvm 1    | 	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> jvm 1    | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> jvm 1    | 	at java.lang.reflect.Method.invoke(Unknown Source)
> jvm 1    | 	at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
> jvm 1    | 	at org.apache.activemq.console.Main.main(Main.java:115)
> jvm 1    | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> jvm 1    | 	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> jvm 1    | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> jvm 1    | 	at java.lang.reflect.Method.invoke(Unknown Source)
> jvm 1    | 	at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
> jvm 1    | 	at java.lang.Thread.run(Unknown Source)
> jvm 1    | Caused by: org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.RuntimeException:
Could not open table 11
> jvm 1    | 	at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:411)
> jvm 1    | 	at org.iq80.leveldb.impl.DbImpl.createWriteBatch(DbImpl.java:707)
> jvm 1    | 	at org.apache.activemq.leveldb.LevelDBClient.replay_from(LevelDBClient.scala:704)
> jvm 1    | 	at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:572)
> jvm 1    | 	at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)
> jvm 1    | 	at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:312)
> jvm 1    | 	at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> jvm 1    | 	at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:653)
> jvm 1    | 	at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:642)
> jvm 1    | 	at org.apache.activemq.broker.BrokerService.start(BrokerService.java:607)
> jvm 1    | 	at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)
> jvm 1    | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> jvm 1    | 	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> jvm 1    | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> jvm 1    | 	at java.lang.reflect.Method.invoke(Unknown Source)
> jvm 1    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1640)
> jvm 1    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1581)
> jvm 1    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1511)
> jvm 1    | 	... 33 more
> jvm 1    | Caused by: java.lang.RuntimeException: Could not open table 11
> jvm 1    | 	at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:87)
> jvm 1    | 	at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:69)
> jvm 1    | 	at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:64)
> jvm 1    | 	at org.iq80.leveldb.util.Level0Iterator.<init>(Level0Iterator.java:29)
> jvm 1    | 	at org.iq80.leveldb.impl.VersionSet.makeInputIterator(VersionSet.java:205)
> jvm 1    | 	at org.iq80.leveldb.impl.DbImpl.doCompactionWork(DbImpl.java:1009)
> jvm 1    | 	at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:478)
> jvm 1    | 	at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:426)
> jvm 1    | 	at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:83)
> jvm 1    | 	at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:396)
> jvm 1    | 	at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:390)
> jvm 1    | 	at java.util.concurrent.FutureTask.run(Unknown Source)
> jvm 1    | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> jvm 1    | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> jvm 1    | 	... 1 more
> jvm 1    | Caused by: java.io.FileNotFoundException: ..\data\leveldb\dirty.index\000011.sst
(Das System kann die angegebene Datei nicht finden)
> jvm 1    | 	at java.io.FileInputStream.open0(Native Method)
> jvm 1    | 	at java.io.FileInputStream.open(Unknown Source)
> jvm 1    | 	at java.io.FileInputStream.<init>(Unknown Source)
> jvm 1    | 	at org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:112)
> jvm 1    | 	at org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:102)
> jvm 1    | 	at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:57)
> jvm 1    | 	at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:54)
> jvm 1    | 	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3579)
> jvm 1    | 	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
> jvm 1    | 	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
> jvm 1    | 	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
> jvm 1    | 	at com.google.common.cache.LocalCache.get(LocalCache.java:3980)
> jvm 1    | 	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3984)
> jvm 1    | 	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4868)
> jvm 1    | 	at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:80)
> jvm 1    | 	... 14 more
> {code}
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message