flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Date Thu, 17 Sep 2015 10:46:15 GMT
Usually that means you are loading different version of the servlet API.
You need to do a bit of classpath troubleshooting to find which jars
contain javax.servlet.AsyncContext and keep just the one included in
Flume/lib

Regards,
Gonzalo

On 17 September 2015 at 11:31, Radu Gheorghe <radu.gheorghe@sematext.com>
wrote:

> Hello flume users!
>
> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
> that with 1.5 but now I seem to fail :(
>
> I got this error in the log (full exception below):
> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
> information does not match signer information of other classes in the same
> package
>
> Basically, what I did was to install Flume 1.6, install Solr 5.3, compile
> Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to Flume's
> classpath. Here's the full command line:
> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
> -Djava.library.path= org.apache.flume.node.Application -n agent -f
> conf/flume-conf.properties
>
> Here's my Java version, if it matters:
> java version "1.7.0_79"
> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>
> From what I can understand out off the Internet (thanks, Internet!) this
> may be caused by multiple jars having different versions of the same thing.
> But I can only find it in Morphlines (via grep -RHls in all the
> classpaths). Maybe I'm bumping into
> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to verify
> this by compiling Kite 0.12.0 but it failed with:
> [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.9:check
> (default) on project kite-data: Too many files with unapproved license: 2
> See RAT report....
>
> So then I compiled only morphlines and that worked, only to bump into the
> same error.
>
> As promised, below is the full exception. For completeness, I'm also
> including the main config and the morphline config:
>
> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  -
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585 counterGroup:{
> name:null counters:{} } } - Exception follows.
> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
> information does not match signer information of other classes in the same
> package
> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
> at
> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
> at
> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
> at
> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
> at
> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> > cat conf/flume-conf.properties
> agent.sources = spoolDir
> agent.channels = spillableMem
> agent.sinks = solr
>
> agent.sources.spoolDir.type = spooldir
> agent.sources.spoolDir.spoolDir = /opt/logs
> agent.sources.spoolDir.batchSize = 1000
> agent.sources.spoolDir.channels = spillableMem
>
> agent.channels.spillableMem.type = SPILLABLEMEMORY
> agent.channels.spillableMem.memoryCapacity = 100000
> agent.channels.spillableMem.overflowCapacity = 1000000
> agent.channels.spillableMem.checkpointDir = /var/run/flume
> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>
> agent.sinks.solr.type =
> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
> agent.sinks.solr.channel = spillableMem
> agent.sinks.solr.morphlineFile = conf/morphline.conf
> agent.sinks.solr.morphlineId = 1
> agent.sinks.solr.batchSize = 10000
>
> > cat conf/morphline.conf
> SOLR_LOCATOR : {
>   collection : gettingstarted
>   solrUrl : "http://192.168.1.3:8983/solr/"
>   batchSize : 5000
> }
>
> morphlines : [
> { id : 1
>  commands : [
>   { readLine { charset : UTF-8 } }
>   {
>     grok {
>      dictionaryFiles : [conf/grok-patterns]
>      expressions : {
>        message : """%{COMBINEDAPACHELOG}"""
>      }
>     }
>   }
> #  { generateUUID {
> #      field : id
> #    }
> #  }
>   {
>     loadSolr {
>       solrLocator : ${SOLR_LOCATOR}
>     }
>   }
>  ]
> }
> ]
>
> I also have a grok-patterns file there, but I don't think it's relevant.
>
> Any ideas are much appreciated!
>
> Thanks and best regards,
> Radu
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>

Mime
View raw message