flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Date Thu, 17 Sep 2015 11:50:14 GMT
That's it.
Remove that jar from the Flume classpath so it doesn't interfere with the
one in Flume (version 2.5)

Note Flume already has the kite libraries. I would just add the integration
with Solr (I suggest you have a look at the version of Flume distributed by
Cloudera)

Gonzalo


On 17 September 2015 at 12:39, Radu Gheorghe <radu.gheorghe@sematext.com>
wrote:

> Hi Gonzalo,
>
> Thanks for your reply. Unfortunately, I've already went that route with no
> success. I can only find out one jar holding that, located in Morphlines:
>
> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all> grep
> -RHls javax.servlet.AsyncContext .
> ./target/lib/javax.servlet-3.0.0.v201112011016.jar
>
> grepping in the Solr and Flume directories got me no result. Do you have
> more ideas?
>
> Best regards,
> Radu
>
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gherreros@gmail.com>
> wrote:
>
>> Usually that means you are loading different version of the servlet API.
>> You need to do a bit of classpath troubleshooting to find which jars
>> contain javax.servlet.AsyncContext and keep just the one included in
>> Flume/lib
>>
>> Regards,
>> Gonzalo
>>
>> On 17 September 2015 at 11:31, Radu Gheorghe <radu.gheorghe@sematext.com>
>> wrote:
>>
>>> Hello flume users!
>>>
>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
>>> that with 1.5 but now I seem to fail :(
>>>
>>> I got this error in the log (full exception below):
>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
>>> information does not match signer information of other classes in the same
>>> package
>>>
>>> Basically, what I did was to install Flume 1.6, install Solr 5.3,
>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to
>>> Flume's classpath. Here's the full command line:
>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>>> conf/flume-conf.properties
>>>
>>> Here's my Java version, if it matters:
>>> java version "1.7.0_79"
>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>>
>>> From what I can understand out off the Internet (thanks, Internet!) this
>>> may be caused by multiple jars having different versions of the same thing.
>>> But I can only find it in Morphlines (via grep -RHls in all the
>>> classpaths). Maybe I'm bumping into
>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>>> verify this by compiling Kite 0.12.0 but it failed with:
>>> [ERROR] Failed to execute goal
>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data:
>>> Too many files with unapproved license: 2 See RAT report....
>>>
>>> So then I compiled only morphlines and that worked, only to bump into
>>> the same error.
>>>
>>> As promised, below is the full exception. For completeness, I'm also
>>> including the main config and the morphline config:
>>>
>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  -
>>> Unable to start SinkRunner: {
>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>>> counterGroup:{ name:null counters:{} } } - Exception follows.
>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
>>> information does not match signer information of other classes in the same
>>> package
>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> at
>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>>> at
>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>>> at
>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>>> at
>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>>> at
>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>>> at
>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>>> at
>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> > cat conf/flume-conf.properties
>>> agent.sources = spoolDir
>>> agent.channels = spillableMem
>>> agent.sinks = solr
>>>
>>> agent.sources.spoolDir.type = spooldir
>>> agent.sources.spoolDir.spoolDir = /opt/logs
>>> agent.sources.spoolDir.batchSize = 1000
>>> agent.sources.spoolDir.channels = spillableMem
>>>
>>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>>> agent.channels.spillableMem.memoryCapacity = 100000
>>> agent.channels.spillableMem.overflowCapacity = 1000000
>>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>>
>>> agent.sinks.solr.type =
>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>>> agent.sinks.solr.channel = spillableMem
>>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>>> agent.sinks.solr.morphlineId = 1
>>> agent.sinks.solr.batchSize = 10000
>>>
>>> > cat conf/morphline.conf
>>> SOLR_LOCATOR : {
>>>   collection : gettingstarted
>>>   solrUrl : "http://192.168.1.3:8983/solr/"
>>>   batchSize : 5000
>>> }
>>>
>>> morphlines : [
>>> { id : 1
>>>  commands : [
>>>   { readLine { charset : UTF-8 } }
>>>   {
>>>     grok {
>>>      dictionaryFiles : [conf/grok-patterns]
>>>      expressions : {
>>>        message : """%{COMBINEDAPACHELOG}"""
>>>      }
>>>     }
>>>   }
>>> #  { generateUUID {
>>> #      field : id
>>> #    }
>>> #  }
>>>   {
>>>     loadSolr {
>>>       solrLocator : ${SOLR_LOCATOR}
>>>     }
>>>   }
>>>  ]
>>> }
>>> ]
>>>
>>> I also have a grok-patterns file there, but I don't think it's relevant.
>>>
>>> Any ideas are much appreciated!
>>>
>>> Thanks and best regards,
>>> Radu
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>
>>
>

Mime
View raw message