activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Timothy Stewart (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMQ-5228) java.lang.NoClassDefFoundError: org/fusesource/leveldbjni/internal/JniDB error during cleanup
Date Sun, 15 Jun 2014 02:34:01 GMT
Timothy Stewart created AMQ-5228:
------------------------------------

             Summary: java.lang.NoClassDefFoundError: org/fusesource/leveldbjni/internal/JniDB
error during cleanup
                 Key: AMQ-5228
                 URL: https://issues.apache.org/jira/browse/AMQ-5228
             Project: ActiveMQ
          Issue Type: Bug
          Components: activemq-leveldb-store
    Affects Versions: 5.10.0, 5.9.1
         Environment: Linux  2.6.32-279.5.2.el6.x86_64 #1 SMP Thu Aug 23 12:05:59 EDT 2012
x86_64 x86_64 x86_64 GNU/Linux

JDK 1.7.0_51

Apache Karaf 2.3.5 with activemq-osgi, activemq-blueprint, activemq-client, activemq-webconsole,
activemq-camel, activemq features installed.  Using version 5.9.1 (and tried 5.10.0)
            Reporter: Timothy Stewart


In our production environment, our storage folder runs out of disk space every few days (75
Gb).  Restarting the container addresses the issue, it cleans it up.  I noticed a stack trace
coming on system.err in the wrapper.log (Karaf starts with a wrapper service).  It may or
may not be related to our disk space issue:

INFO   | jvm 1    | 2014/06/13 03:22:36 | Exception in thread "Thread-117" java.lang.NoClassDefFoundError:
org/fusesource/leveldbjni/internal/JniDB
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.activemq.leveldb.LevelDBClient$RichDB.compact(LevelDBClient.scala:377)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.activemq.leveldb.LevelDBClient.gc(LevelDBClient.scala:1647)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.activemq.leveldb.DBManager$$anonfun$pollGc$1$$anonfun$apply$mcV$sp$2.apply$mcV$sp(DBManager.scala:648)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at java.lang.Thread.run(Thread.java:744)
INFO   | jvm 1    | 2014/06/13 03:22:36 | Caused by: java.lang.ClassNotFoundException: org.fusesource.leveldbjni.internal.JniDB
not found by org.apache.activemq.activemq-osgi [105]
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
INFO   | jvm 1    | 2014/06/13 03:22:36 |       ... 7 more


I see the same problem in our dev environment but could not replicate it.  I was finally able
to replicate by using the hawtio console to execute the compact operation.  Everytime I do
this, the same stack trace outputs and the operation cycles endlessly (well as long as I've
waited anyhow).  The 5.10.0 stack trace when I execute the operation is:

INFO   | jvm 2    | 2014/06/14 21:00:44 | Exception in thread "Thread-106" java.lang.NoClassDefFoundError:
org/fusesource/leveldbjni/internal/JniDB
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.activemq.leveldb.LevelDBClient$RichDB.compact(LevelDBClient.scala:378)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.activemq.leveldb.LevelDBClient.gc(LevelDBClient.scala:1654)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.activemq.leveldb.LevelDBStoreView$$anonfun$compact$1.apply$mcV$sp(LevelDBStore.scala:126)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at java.lang.Thread.run(Thread.java:744)
INFO   | jvm 2    | 2014/06/14 21:00:44 | Caused by: java.lang.ClassNotFoundException: org.fusesource.leveldbjni.internal.JniDB
not found by org.apache.activemq.activemq-osgi [390]
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
INFO   | jvm 2    | 2014/06/14 21:00:44 |       ... 7 more


I started looking at the activemq-osgi and activemq-leveldb project poms, and noticed that
the leveldbjni projects are commented out with a note that we want to focus on hardening the
pure java version.  With that in mind, I switched the indexFactory on my leveldb config to
indexFactory="org.iq80.leveldb.impl.Iq80DBFactory".  I restarted, and tried the compact operation
again, but got the same stacktrace.

My leveldb config now looks like:

<amq:persistenceAdapter>
            <amq:levelDB directory="[[karaf.storage]]/activemq/default/leveldb" logSize="107374182"
logCompression="snappy" indexFactory="org.iq80.leveldb.impl.Iq80DBFactory" />
        </amq:persistenceAdapter>

The logCompression and the indexFactory were both added as an attempt to mitigate the problem,
but the issue exists either way.  I also saw a note on the replicated levedb store about scheduled
jobs, which are used a lot, but when I looked at those folders in our storage folder it seems
to use a completely different kahadb store, so I threw that out as a reason.

I did attempt to add leveldbjni-linux64 or leveldbjni-all to the osgi deployment, but it does
not export the internal package which activemq-osgi wants to import.  I could build a new
package of these, or rebuild activemq-osgi with the commented dependencies removed.. perhaps
that will work, but it seems that as it is with the dependencies commented out that there
is something broken within the compact routine.





--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message