Mailing-List: contact commits-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@activemq.apache.org
Date: Wed, 9 Jan 2013 07:02:14 +0000 (UTC)
From: "Lionel Cons (JIRA)" <jira@apache.org>
To: commits@activemq.apache.org
Message-ID: <JIRA.12624775.1355986502932.102297.1357714934082@arcas>
In-Reply-To: <JIRA.12624775.1355986502932@arcas>
References: <JIRA.12624775.1355986502932@arcas>
Subject: [jira] [Commented] (APLO-282) org.iq80.leveldb.DBException -
 possible LevelDB store corruption?
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/APLO-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13547702#comment-13547702 ] 

Lionel Cons commented on APLO-282:
----------------------------------

Correct, the store does not contain any 016282.sst file.

Regarding recovery, yes, Apollo should have a way to (attempt to) recover data. In that case, I had to remove completely the data store, loosing everything. It would be nice to have recovery like KahaDB has (http://fusesource.com/docs/broker/5.5/persistence/KahaDB-Recovery.html). FWIW, I have archived the (corrupted) data store in case it helps testing recovery.

It is not clear what caused the problem. The disks were not full but there might have been a disk related problem as the disks are on SAN.
                
> org.iq80.leveldb.DBException - possible LevelDB store corruption?
> -----------------------------------------------------------------
>
>                 Key: APLO-282
>                 URL: https://issues.apache.org/jira/browse/APLO-282
>             Project: ActiveMQ Apollo
>          Issue Type: Bug
>         Environment: apollo-99-trunk-20121105.032511-125
>            Reporter: Lionel Cons
>
> One of our broker refuses to start and logs:
> 2012-12-19 17:15:27,828 | INFO  | Starting store: leveldb store at /var/lib/apollo/data | 
> 2012-12-19 17:15:28,001 | INFO  | Opening the log file took: 62.94 ms | 
> 2012-12-19 17:15:28,214 | WARN  | DB operation failed. (entering recovery mode): org.iq80.leveldb.DBException: IO error: /var/lib/apollo/data/dirty.index/016282.sst: No such file or directory | 13bb3f2b7c0
> [...]
> 2012-12-19 17:15:32,863 | INFO  | broker startup is waiting on start virtual-host: apollo | 
> and it hangs there.
> Here is the stack trace:
> 2012-12-19 17:15:28,217 | WARN  | stackref=13bb3f2b7c0
> org.iq80.leveldb.DBException: IO error: /var/lib/apollo/data/dirty.index/016282.sst: No such file or directory
> 	at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:78)
> 	at org.apache.activemq.apollo.broker.store.leveldb.HelperTrait$RichDB.get(HelperTrait.scala:113)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.org$apache$activemq$apollo$broker$store$leveldb$LevelDBClient$$load_log_refs(LevelDBClient.scala:595)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply$mcV$sp(LevelDBClient.scala:356)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply(LevelDBClient.scala:332)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply(LevelDBClient.scala:332)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry(LevelDBClient.scala:714)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.start(LevelDBClient.scala:332)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBStore$$anonfun$_start$4.apply$mcV$sp(LevelDBStore.scala:114)
> 	at org.fusesource.hawtdispatch.package$$anon$3.run(hawtdispatch.scala:357)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/lib/apollo/data/dirty.index/016282.sst: No such file or directory
> 	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> 	at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:307)
> 	at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:300)
> 	at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:293)
> 	at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:73)
> 	... 12 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira