flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radu Gheorghe <radu.gheor...@sematext.com>
Subject Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Date Thu, 17 Sep 2015 10:31:26 GMT
Hello flume users!

I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did that
with 1.5 but now I seem to fail :(

I got this error in the log (full exception below):
java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
information does not match signer information of other classes in the same
package

Basically, what I did was to install Flume 1.6, install Solr 5.3, compile
Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to Flume's
classpath. Here's the full command line:
exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
-Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
'/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
-Djava.library.path= org.apache.flume.node.Application -n agent -f
conf/flume-conf.properties

Here's my Java version, if it matters:
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

>From what I can understand out off the Internet (thanks, Internet!) this
may be caused by multiple jars having different versions of the same thing.
But I can only find it in Morphlines (via grep -RHls in all the
classpaths). Maybe I'm bumping into
https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to verify
this by compiling Kite 0.12.0 but it failed with:
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.9:check
(default) on project kite-data: Too many files with unapproved license: 2
See RAT report....

So then I compiled only morphlines and that worked, only to bump into the
same error.

As promised, below is the full exception. For completeness, I'm also
including the main config and the morphline config:

17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
(org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  -
Unable to start SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585 counterGroup:{
name:null counters:{} } } - Exception follows.
java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
information does not match signer information of other classes in the same
package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
at
org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
at
org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
at
org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
at
org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
at
org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


> cat conf/flume-conf.properties
agent.sources = spoolDir
agent.channels = spillableMem
agent.sinks = solr

agent.sources.spoolDir.type = spooldir
agent.sources.spoolDir.spoolDir = /opt/logs
agent.sources.spoolDir.batchSize = 1000
agent.sources.spoolDir.channels = spillableMem

agent.channels.spillableMem.type = SPILLABLEMEMORY
agent.channels.spillableMem.memoryCapacity = 100000
agent.channels.spillableMem.overflowCapacity = 1000000
agent.channels.spillableMem.checkpointDir = /var/run/flume
agent.channels.spillableMem.dataDirs = /var/run/flume-data

agent.sinks.solr.type =
org.apache.flume.sink.solr.morphline.MorphlineSolrSink
agent.sinks.solr.channel = spillableMem
agent.sinks.solr.morphlineFile = conf/morphline.conf
agent.sinks.solr.morphlineId = 1
agent.sinks.solr.batchSize = 10000

> cat conf/morphline.conf
SOLR_LOCATOR : {
  collection : gettingstarted
  solrUrl : "http://192.168.1.3:8983/solr/"
  batchSize : 5000
}

morphlines : [
{ id : 1
 commands : [
  { readLine { charset : UTF-8 } }
  {
    grok {
     dictionaryFiles : [conf/grok-patterns]
     expressions : {
       message : """%{COMBINEDAPACHELOG}"""
     }
    }
  }
#  { generateUUID {
#      field : id
#    }
#  }
  {
    loadSolr {
      solrLocator : ${SOLR_LOCATOR}
    }
  }
 ]
}
]

I also have a grok-patterns file there, but I don't think it's relevant.

Any ideas are much appreciated!

Thanks and best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

Mime
View raw message