activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-5228) java.lang.NoClassDefFoundError: org/fusesource/leveldbjni/internal/JniDB error during cleanup
Date Mon, 21 Sep 2015 09:47:05 GMT

    [ https://issues.apache.org/jira/browse/AMQ-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900454#comment-14900454
] 

Stefan commented on AMQ-5228:
-----------------------------

Any news here? Right now our LevelDB is growing as well and the only way to circumvent is
to wait until all queues are empty, shut-down ActiveMQ and delete the files manually. Without
the compact, its not usable so we switched back to kahaDB

> java.lang.NoClassDefFoundError: org/fusesource/leveldbjni/internal/JniDB error during
cleanup
> ---------------------------------------------------------------------------------------------
>
>                 Key: AMQ-5228
>                 URL: https://issues.apache.org/jira/browse/AMQ-5228
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: activemq-leveldb-store
>    Affects Versions: 5.9.1, 5.10.0
>         Environment: Linux  2.6.32-279.5.2.el6.x86_64 #1 SMP Thu Aug 23 12:05:59 EDT
2012 x86_64 x86_64 x86_64 GNU/Linux
> JDK 1.7.0_51
> Apache Karaf 2.3.5 with activemq-osgi, activemq-blueprint, activemq-client, activemq-webconsole,
activemq-camel, activemq features installed.  Using version 5.9.1 (and tried 5.10.0)
>            Reporter: Timothy Stewart
>
> In our production environment, our storage folder runs out of disk space every few days
(75 Gb).  Restarting the container addresses the issue, it cleans it up.  I noticed a stack
trace coming on system.err in the wrapper.log (Karaf starts with a wrapper service).  It may
or may not be related to our disk space issue:
> INFO   | jvm 1    | 2014/06/13 03:22:36 | Exception in thread "Thread-117" java.lang.NoClassDefFoundError:
org/fusesource/leveldbjni/internal/JniDB
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.activemq.leveldb.LevelDBClient$RichDB.compact(LevelDBClient.scala:377)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.activemq.leveldb.LevelDBClient.gc(LevelDBClient.scala:1647)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.activemq.leveldb.DBManager$$anonfun$pollGc$1$$anonfun$apply$mcV$sp$2.apply$mcV$sp(DBManager.scala:648)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at java.lang.Thread.run(Thread.java:744)
> INFO   | jvm 1    | 2014/06/13 03:22:36 | Caused by: java.lang.ClassNotFoundException:
org.fusesource.leveldbjni.internal.JniDB not found by org.apache.activemq.activemq-osgi [105]
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> INFO   | jvm 1    | 2014/06/13 03:22:36 |       ... 7 more
> I see the same problem in our dev environment but could not replicate it.  I was finally
able to replicate by using the hawtio console to execute the compact operation.  Everytime
I do this, the same stack trace outputs and the operation cycles endlessly (well as long as
I've waited anyhow).  The 5.10.0 stack trace when I execute the operation is:
> INFO   | jvm 2    | 2014/06/14 21:00:44 | Exception in thread "Thread-106" java.lang.NoClassDefFoundError:
org/fusesource/leveldbjni/internal/JniDB
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.activemq.leveldb.LevelDBClient$RichDB.compact(LevelDBClient.scala:378)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.activemq.leveldb.LevelDBClient.gc(LevelDBClient.scala:1654)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.activemq.leveldb.LevelDBStoreView$$anonfun$compact$1.apply$mcV$sp(LevelDBStore.scala:126)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at java.lang.Thread.run(Thread.java:744)
> INFO   | jvm 2    | 2014/06/14 21:00:44 | Caused by: java.lang.ClassNotFoundException:
org.fusesource.leveldbjni.internal.JniDB not found by org.apache.activemq.activemq-osgi [390]
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> INFO   | jvm 2    | 2014/06/14 21:00:44 |       ... 7 more
> I started looking at the activemq-osgi and activemq-leveldb project poms, and noticed
that the leveldbjni projects are commented out with a note that we want to focus on hardening
the pure java version.  With that in mind, I switched the indexFactory on my leveldb config
to indexFactory="org.iq80.leveldb.impl.Iq80DBFactory".  I restarted, and tried the compact
operation again, but got the same stacktrace.
> My leveldb config now looks like:
> <amq:persistenceAdapter>
>             <amq:levelDB directory="[[karaf.storage]]/activemq/default/leveldb" logSize="107374182"
logCompression="snappy" indexFactory="org.iq80.leveldb.impl.Iq80DBFactory" />
>         </amq:persistenceAdapter>
> The logCompression and the indexFactory were both added as an attempt to mitigate the
problem, but the issue exists either way.  I also saw a note on the replicated levedb store
about scheduled jobs, which are used a lot, but when I looked at those folders in our storage
folder it seems to use a completely different kahadb store, so I threw that out as a reason.
> I did attempt to add leveldbjni-linux64 or leveldbjni-all to the osgi deployment, but
it does not export the internal package which activemq-osgi wants to import.  I could build
a new package of these, or rebuild activemq-osgi with the commented dependencies removed..
perhaps that will work, but it seems that as it is with the dependencies commented out that
there is something broken within the compact routine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message