activemq-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lionel Cons (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APLO-282) org.iq80.leveldb.DBException - possible LevelDB store corruption?
Date Wed, 30 Jan 2013 13:33:13 GMT

    [ https://issues.apache.org/jira/browse/APLO-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566470#comment-13566470
] 

Lionel Cons commented on APLO-282:
----------------------------------

We still have problems even with a recent snapshot (apollo-99-trunk-20130129.042953-170).
Here is what we got:

2013-01-30 13:16:19,560 | WARN  | DB operation failed. (entering recovery mode): org.iq80.leveldb.DBException:
IO error: /var/lib/apollo/data/dirty.index/008152.sst: No such file or directory | 13c8b62c13f
2013-01-30 13:20:26,575 | WARN  | HTTP request from '137.138.34.25' for GET '/broker/config/files.html'
caused internal server error: java.lang.AssertionError: assertion failed: You must not do
any blocking waits when executing on HawtDispatch thread pool. | 13c8b62c140

And the corresponding stack traces:

2013-01-30 13:16:19,561 | WARN  | stackref=13c8b62c13f
org.iq80.leveldb.DBException: IO error: /var/lib/apollo/data/dirty.index/008152.sst: No such
file or directory
	at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:78)
	at org.apache.activemq.apollo.broker.store.leveldb.HelperTrait$RichDB.get(HelperTrait.scala:113)
	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.org$apache$activemq$apollo$broker$store$leveldb$LevelDBClient$$load_log_refs(LevelDBClient.scala:596)
	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply$mcV$sp(LevelDBClient.scala:357)
	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply(LevelDBClient.scala:333)
	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply(LevelDBClient.scala:333)
	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry(LevelDBClient.scala:715)
	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.start(LevelDBClient.scala:333)
	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBStore$$anonfun$_start$4.apply$mcV$sp(LevelDBStore.scala:117)
	at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/lib/apollo/data/dirty.index/008152.sst:
No such file or directory
	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
	at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:307)
	at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:300)
	at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:293)
	at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:73)
	... 12 more

2013-01-30 13:20:26,580 | WARN  | stackref=13c8b62c140
java.lang.AssertionError: assertion failed: You must not do any blocking waits when executing
on HawtDispatch thread pool.
	at scala.Predef$.assert(Predef.scala:179)
	at org.fusesource.hawtdispatch.SettableFuture$class.await(Future.scala:63)
	at org.fusesource.hawtdispatch.Future$$anon$1.await(Future.scala:117)
	at org.apache.activemq.apollo.util.package$FutureResult$.unwrap_future_result(util.scala:40)
	at org.apache.activemq.apollo.web.resources.ConfigurationResource.if_allowed(ConfigurationResource.scala:72)
	at org.apache.activemq.apollo.web.resources.ConfigurationResource.list(ConfigurationResource.scala:106)
	at org.apache.activemq.apollo.web.resources.ConfigurationResource$$anonfun$list_html$1.apply(ConfigurationResource.scala:121)
	at org.apache.activemq.apollo.web.resources.ConfigurationResource$$anonfun$list_html$1.apply(ConfigurationResource.scala:121)
	at org.apache.activemq.apollo.web.resources.ConfigurationResource$$anonfun$if_allowed$1$$anonfun$apply$2$$anonfun$apply$3.apply(ConfigurationResource.scala:75)
	at org.apache.activemq.apollo.web.resources.ConfigurationResource$$anonfun$if_allowed$1$$anonfun$apply$2$$anonfun$apply$3.apply(ConfigurationResource.scala:75)
	at org.apache.activemq.apollo.util.package$$anonfun$sync$1.apply$mcV$sp(util.scala:59)
	at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
	at org.fusesource.hawtdispatch.internal.SerialDispatchQueue.run(SerialDispatchQueue.java:96)
	at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:77)

                
> org.iq80.leveldb.DBException - possible LevelDB store corruption?
> -----------------------------------------------------------------
>
>                 Key: APLO-282
>                 URL: https://issues.apache.org/jira/browse/APLO-282
>             Project: ActiveMQ Apollo
>          Issue Type: Bug
>         Environment: apollo-99-trunk-20121105.032511-125
>            Reporter: Lionel Cons
>            Assignee: Hiram Chirino
>             Fix For: 1.6
>
>
> One of our broker refuses to start and logs:
> 2012-12-19 17:15:27,828 | INFO  | Starting store: leveldb store at /var/lib/apollo/data
| 
> 2012-12-19 17:15:28,001 | INFO  | Opening the log file took: 62.94 ms | 
> 2012-12-19 17:15:28,214 | WARN  | DB operation failed. (entering recovery mode): org.iq80.leveldb.DBException:
IO error: /var/lib/apollo/data/dirty.index/016282.sst: No such file or directory | 13bb3f2b7c0
> [...]
> 2012-12-19 17:15:32,863 | INFO  | broker startup is waiting on start virtual-host: apollo
| 
> and it hangs there.
> Here is the stack trace:
> 2012-12-19 17:15:28,217 | WARN  | stackref=13bb3f2b7c0
> org.iq80.leveldb.DBException: IO error: /var/lib/apollo/data/dirty.index/016282.sst:
No such file or directory
> 	at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:78)
> 	at org.apache.activemq.apollo.broker.store.leveldb.HelperTrait$RichDB.get(HelperTrait.scala:113)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.org$apache$activemq$apollo$broker$store$leveldb$LevelDBClient$$load_log_refs(LevelDBClient.scala:595)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply$mcV$sp(LevelDBClient.scala:356)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply(LevelDBClient.scala:332)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply(LevelDBClient.scala:332)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry(LevelDBClient.scala:714)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.start(LevelDBClient.scala:332)
> 	at org.apache.activemq.apollo.broker.store.leveldb.LevelDBStore$$anonfun$_start$4.apply$mcV$sp(LevelDBStore.scala:114)
> 	at org.fusesource.hawtdispatch.package$$anon$3.run(hawtdispatch.scala:357)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/lib/apollo/data/dirty.index/016282.sst:
No such file or directory
> 	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> 	at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:307)
> 	at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:300)
> 	at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:293)
> 	at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:73)
> 	... 12 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message