Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8BAC717E05 for ; Thu, 17 Sep 2015 13:21:29 +0000 (UTC) Received: (qmail 66523 invoked by uid 500); 17 Sep 2015 13:21:29 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 66464 invoked by uid 500); 17 Sep 2015 13:21:29 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 66452 invoked by uid 99); 17 Sep 2015 13:21:29 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Sep 2015 13:21:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id B5D891A21AC for ; Thu, 17 Sep 2015 13:21:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.982 X-Spam-Level: *** X-Spam-Status: No, score=3.982 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id jPIIsw19Qk3C for ; Thu, 17 Sep 2015 13:21:25 +0000 (UTC) Received: from mail-wi0-f179.google.com (mail-wi0-f179.google.com [209.85.212.179]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 6B93520864 for ; Thu, 17 Sep 2015 13:21:24 +0000 (UTC) Received: by wiclk2 with SMTP id lk2so23671339wic.0 for ; Thu, 17 Sep 2015 06:21:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=0OQqQyfScY2BOFAlspisofwg/oZPwvN67etRSByQenI=; b=YaSxlxSlgrUlCxlV1g10CQ5ggBAhb+T1CK9RdRog84zNB0/pOTLg7Y+mlbWlmYjWta pVm9OsxteMoUEi2biW+bquKHBDPXxOjQhQpwdh1KMFQVIoINLKx6o/2ZbyXoY8tOXX74 NsD9cYIO8z0pVzvMYRb4LkfsJEcajdrV/m29s8X6Iejsujpd367IgNobTlKKsKwy4ImI wWQKAIEJH7NkO90fSNKeYO/VFejzRJn9kTHa1aWFKSSDPuGtZrw7XAww/VwrPVXcwrri t4Gx48OmCjzb6eDiyQO/cxTsJKEgEEpNTGYIem3JHABXzGZOFUpUz9yhVHHdroU7SsF3 fEhg== X-Gm-Message-State: ALoCoQnm42q5+2WlD1bduh5mQqYjFgJcF7yAxlpM93SlEDVUH2jnst3W5+3pwYoshgWHnDDOJcP5 MIME-Version: 1.0 X-Received: by 10.180.188.211 with SMTP id gc19mr8123366wic.81.1442496082991; Thu, 17 Sep 2015 06:21:22 -0700 (PDT) Received: by 10.28.0.130 with HTTP; Thu, 17 Sep 2015 06:21:22 -0700 (PDT) In-Reply-To: References: Date: Thu, 17 Sep 2015 16:21:22 +0300 Message-ID: Subject: Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink From: Radu Gheorghe To: user@flume.apache.org Content-Type: multipart/alternative; boundary=001a11c3825a9df7b1051ff14af7 --001a11c3825a9df7b1051ff14af7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks, Gonzalo! That actually got me further. A Solr exception later (got around it by removing 4.10 Solr libs which I assume don't work with my 5.3 for which I already have libs in CP), then I got another error pointing the same sort of issue. So I just went ahead and removed all duplicated jars (though many were din different versions) from kite. With something as ugly as this: for lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls /opt/apache-flume-1.6.0-bin/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm $lib*; fi; done Then Flume complained about some codahale metrics class missing, so I moved back the metrics-core library to kite. And now I get this: java.lang.VerifyError: Cannot inherit from final class I'm not sure if you need the whole stacktrace (I will paste it below) but I have again the suspicion of duplicated stuff or mismatched versions. I'm not sure I'm on the right path, so you were pointing me to different ones: - add the integration with Solr. Can you tell me which are the jars needed? >From which subproject? - look at the version of Flume distributed by Cloudera. I can't find just Flume, I suppose I need to get the whole CDH or something like that? Thanks again for all your help! Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Herreros wrote: > That's it. > Remove that jar from the Flume classpath so it doesn't interfere with the > one in Flume (version 2.5) > > Note Flume already has the kite libraries. I would just add the > integration with Solr (I suggest you have a look at the version of Flume > distributed by Cloudera) > > Gonzalo > > > On 17 September 2015 at 12:39, Radu Gheorghe > wrote: > >> Hi Gonzalo, >> >> Thanks for your reply. Unfortunately, I've already went that route with >> no success. I can only find out one jar holding that, located in Morphli= nes: >> >> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all> >> grep -RHls javax.servlet.AsyncContext . >> ./target/lib/javax.servlet-3.0.0.v201112011016.jar >> >> grepping in the Solr and Flume directories got me no result. Do you have >> more ideas? >> >> Best regards, >> Radu >> >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros >> wrote: >> >>> Usually that means you are loading different version of the servlet API= . >>> You need to do a bit of classpath troubleshooting to find which jars >>> contain javax.servlet.AsyncContext and keep just the one included in >>> Flume/lib >>> >>> Regards, >>> Gonzalo >>> >>> On 17 September 2015 at 11:31, Radu Gheorghe >> > wrote: >>> >>>> Hello flume users! >>>> >>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did >>>> that with 1.5 but now I seem to fail :( >>>> >>>> I got this error in the log (full exception below): >>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s >>>> signer information does not match signer information of other classes = in >>>> the same package >>>> >>>> Basically, what I did was to install Flume 1.6, install Solr 5.3, >>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars t= o >>>> Flume's classpath. Here's the full command line: >>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m >>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=3Dtrue -cp >>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/o= pt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/ho= me/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/li= b/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*' >>>> -Djava.library.path=3D org.apache.flume.node.Application -n agent -f >>>> conf/flume-conf.properties >>>> >>>> Here's my Java version, if it matters: >>>> java version "1.7.0_79" >>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64) >>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) >>>> >>>> From what I can understand out off the Internet (thanks, Internet!) >>>> this may be caused by multiple jars having different versions of the s= ame >>>> thing. But I can only find it in Morphlines (via grep -RHls in all the >>>> classpaths). Maybe I'm bumping into >>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to >>>> verify this by compiling Kite 0.12.0 but it failed with: >>>> [ERROR] Failed to execute goal >>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-d= ata: >>>> Too many files with unapproved license: 2 See RAT report.... >>>> >>>> So then I compiled only morphlines and that worked, only to bump into >>>> the same error. >>>> >>>> As promised, below is the full exception. For completeness, I'm also >>>> including the main config and the morphline config: >>>> >>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8] >>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:25= 3) - >>>> Unable to start SinkRunner: { >>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585 >>>> counterGroup:{ name:null counters:{} } } - Exception follows. >>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s >>>> signer information does not match signer information of other classes = in >>>> the same package >>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) >>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) >>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794) >>>> at >>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142= ) >>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) >>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>> at java.lang.ClassLoader.defineClass1(Native Method) >>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800) >>>> at >>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142= ) >>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) >>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>> at >>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$Class= Info.load(ClassPath.java:224) >>>> at >>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(Morphlin= eContext.java:176) >>>> at >>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(Morph= lineContext.java:91) >>>> at org.kitesdk.morphline.stdlib.Pipe.(Pipe.java:43) >>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40) >>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126) >>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55) >>>> at >>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(Mo= rphlineHandlerImpl.java:101) >>>> at >>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink= .java:97) >>>> at >>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.= java:46) >>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79) >>>> at >>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(Lif= ecycleSupervisor.java:251) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471= ) >>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.a= ccess$301(ScheduledThreadPoolExecutor.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.r= un(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> >>>> > cat conf/flume-conf.properties >>>> agent.sources =3D spoolDir >>>> agent.channels =3D spillableMem >>>> agent.sinks =3D solr >>>> >>>> agent.sources.spoolDir.type =3D spooldir >>>> agent.sources.spoolDir.spoolDir =3D /opt/logs >>>> agent.sources.spoolDir.batchSize =3D 1000 >>>> agent.sources.spoolDir.channels =3D spillableMem >>>> >>>> agent.channels.spillableMem.type =3D SPILLABLEMEMORY >>>> agent.channels.spillableMem.memoryCapacity =3D 100000 >>>> agent.channels.spillableMem.overflowCapacity =3D 1000000 >>>> agent.channels.spillableMem.checkpointDir =3D /var/run/flume >>>> agent.channels.spillableMem.dataDirs =3D /var/run/flume-data >>>> >>>> agent.sinks.solr.type =3D >>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink >>>> agent.sinks.solr.channel =3D spillableMem >>>> agent.sinks.solr.morphlineFile =3D conf/morphline.conf >>>> agent.sinks.solr.morphlineId =3D 1 >>>> agent.sinks.solr.batchSize =3D 10000 >>>> >>>> > cat conf/morphline.conf >>>> SOLR_LOCATOR : { >>>> collection : gettingstarted >>>> solrUrl : "http://192.168.1.3:8983/solr/" >>>> batchSize : 5000 >>>> } >>>> >>>> morphlines : [ >>>> { id : 1 >>>> commands : [ >>>> { readLine { charset : UTF-8 } } >>>> { >>>> grok { >>>> dictionaryFiles : [conf/grok-patterns] >>>> expressions : { >>>> message : """%{COMBINEDAPACHELOG}""" >>>> } >>>> } >>>> } >>>> # { generateUUID { >>>> # field : id >>>> # } >>>> # } >>>> { >>>> loadSolr { >>>> solrLocator : ${SOLR_LOCATOR} >>>> } >>>> } >>>> ] >>>> } >>>> ] >>>> >>>> I also have a grok-patterns file there, but I don't think it's relevan= t. >>>> >>>> Any ideas are much appreciated! >>>> >>>> Thanks and best regards, >>>> Radu >>>> -- >>>> Performance Monitoring * Log Analytics * Search Analytics >>>> Solr & Elasticsearch Support * http://sematext.com/ >>>> >>> >>> >> > --001a11c3825a9df7b1051ff14af7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks, Gonzalo! That actually got me further. A Solr= exception later (got around it by removing 4.10 Solr libs which I assume d= on't work with my 5.3 for which I already have libs in CP), then I got = another error pointing the same sort of issue.

So I just went ahead and removed all duplicated jars (though many were din= different versions) from kite. With something as ugly as this:
f= or lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls /opt/apache-flume-1.6.0-bi= n/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm $lib*; fi; done

Then Flume complained about some codahale metrics cl= ass missing, so I moved back the metrics-core library to kite. And now I ge= t this:

java.lang.VerifyError: Cannot inherit from= final class

I'm not sure if you need the = whole stacktrace (I will paste it below) but I have again the suspicion of = duplicated stuff or mismatched versions.

I'm n= ot sure I'm on the right path, so you were pointing me to different one= s:
- add the integration with Solr. Can you tell me which are the= jars needed? From which subproject?
- look at the version of Flu= me distributed by Cloudera. I can't find just Flume, I suppose I need t= o get the whole CDH or something like that?

Thanks= again for all your help!

Best regards,
= Radu

--
Performance Monito= ring * Log Analytics * Search Analytics
Solr & Elasticsearch Support *=C2=A0= http://sematext.com/

On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Her= reros <gherreros@gmail.com> wrote:
That's it.
Remove that jar from the = Flume classpath so it doesn't interfere with the one in Flume (version = 2.5)

Note Flume already has the kite libraries. I = would just add the integration with Solr (I suggest you have a look at the = version of Flume distributed by Cloudera)

Gonzalo

<= /span>

On 17 September 2015 at 12:39, Radu Gheo= rghe <radu.gheorghe@sematext.com> wrote:
Hi Gonzalo,

Tha= nks for your reply. Unfortunately, I've already went that route with no= success. I can only find out one jar holding that, located in Morphlines:<= /div>

radu@rgheorghe-suse:~/gits/kite/kite-morphlin= es/kite-morphlines-all> grep -RHls javax.servlet.AsyncContext .
./target/lib/javax.servlet-3.0.0.v201112011016.jar

grepping in the Solr and Flume directories got me no result. Do yo= u have more ideas?

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Sear= ch Analytics
Solr & Elasticsearch Support *=C2=A0http://sematext.com/

On Thu, Sep 17, 2015 at 1:4= 6 PM, Gonzalo Herreros <gherreros@gmail.com> wrote:
Usually that means you are loa= ding different version of the servlet API.
You need to do a bit of clas= spath troubleshooting to find which jars contain=C2=A0javax.servlet.AsyncContext and keep just the one in= cluded in Flume/lib

R= egards,
Gonza= lo

On 17 September 2015 at 11:31, Radu Gheorghe <radu.= gheorghe@sematext.com> wrote:
Hello flume users!

I'm trying to= write log to Solr using Flume's MorphlineSolrSink. I did that with 1.5= but now I seem to fail :(

I got this error in the= log (full exception below):
java.lang.SecurityException: class &= quot;javax.servlet.AsyncContext"'s signer information does not mat= ch signer information of other classes in the same package
Basically, what I did was to install Flume 1.6, install Solr 5= .3, compile Kite SDK 1.1.0 and then added Solr and "morphlines-all&quo= t; jars to Flume's classpath. Here's the full command line:
exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m -Dcom.sun.manageme= nt.jmxremote -Dmetadata-complete=3Dtrue -cp '/opt/apache-flume-1.6.0-bi= n/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/= opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphline= s/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/= solr-5.3.0/dist/solrj-lib/*:/lib/*' -Djava.library.path=3D org.apache.f= lume.node.Application -n agent -f conf/flume-conf.properties
=
Here's my Java version, if it matters:
ja= va version "1.7.0_79"
OpenJDK Runtime Environment (Iced= Tea 2.5.5) (suse-7.4-x86_64)
OpenJDK 64-Bit Server VM (build 24.7= 9-b02, mixed mode)

From what I can understan= d out off the Internet (thanks, Internet!) this may be caused by multiple j= ars having different versions of the same thing. But I can only find it in = Morphlines (via grep -RHls in all the classpaths). Maybe I'm bumping in= to=C2=A0https://issues.apache.org/jira/browse/FLUME-2448 ? I was tr= ying to verify this by compiling Kite 0.12.0 but it failed with:
= [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.9:check (= default) on project kite-data: Too many files with unapproved license: 2 Se= e RAT report....

So then I compiled only morph= lines and that worked, only to bump into the same error.

As promised, below is the full exception. For completeness, I'm = also including the main config and the morphline config:

17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8] (org.a= pache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) =C2=A0- = Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProce= ssor@41a72585 counterGroup:{ name:null counters:{} } } - Exception follows.=
java.lang.SecurityException: class "javax.servlet.AsyncCont= ext"'s signer information does not match signer information of oth= er classes in the same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
<= div> at java.lang.ClassLoader.p= reDefineClass(ClassLoader.java:666)
at java.lang.ClassLoader.defineClass(ClassLoader.java:794)<= /div>
at java.security.Sec= ureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(U= RLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.= run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessContr= oller.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)=
at java.lang.ClassL= oader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.ja= va:308)
at java.lang= .ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Metho= d)
at java.lang.Clas= sLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(Secure= ClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.ac= cess$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoad= er$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)=
at java.net.URLClas= sLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:= 425)
at sun.misc.Lau= ncher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoade= r.java:358)
at org.k= itesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(= ClassPath.java:224)
= at org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineC= ontext.java:176)
at = org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineC= ontext.java:91)
at o= rg.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
at org.kitesdk.morphline.stdlib.Pip= eBuilder.build(PipeBuilder.java:40)
at org.kitesdk.morphline.base.Compiler.compile(Compiler.jav= a:126)
at org.kitesd= k.morphline.base.Compiler.compile(Compiler.java:55)
at org.apache.flume.sink.solr.morphline.M= orphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
at org.apache.flume.sink.solr.morp= hline.MorphlineSink.start(MorphlineSink.java:97)
at org.apache.flume.sink.DefaultSinkProcessor.= start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.li= fecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:25= 1)
at java.util.conc= urrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.ru= nAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFut= ureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThrea= dPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)=
at java.util.concur= rent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
<= span style=3D"white-space:pre-wrap"> at java.util.concurrent.ThreadP= oolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)<= /div>


> cat conf/flume-conf= .properties
agent.sources =3D spoolDir
agent.channels = =3D spillableMem
agent.sinks =3D solr

ag= ent.sources.spoolDir.type =3D spooldir
agent.sources.spoolDir.spo= olDir =3D /opt/logs
agent.sources.spoolDir.batchSize =3D 1000
agent.sources.spoolDir.channels =3D spillableMem

agent.channels.spillableMem.type =3D SPILLABLEMEMORY
agent.= channels.spillableMem.memoryCapacity =3D 100000
agent.channels.sp= illableMem.overflowCapacity =3D 1000000
agent.channels.spillableM= em.checkpointDir =3D /var/run/flume
agent.channels.spillableMem.d= ataDirs =3D /var/run/flume-data

agent.sinks.solr.t= ype =3D org.apache.flume.sink.solr.morphline.MorphlineSolrSink
ag= ent.sinks.solr.channel =3D spillableMem
agent.sinks.solr.morphlin= eFile =3D conf/morphline.conf
agent.sinks.solr.morphlineId =3D 1<= /div>
agent.sinks.solr.batchSize =3D 10000

> cat conf/morphline.conf=C2=A0
SOLR_LOCATOR : {
=
=C2=A0 collection : gettingstarted
=C2=A0 solrUrl : "http://192.168.1.= 3:8983/solr/"
=C2=A0 batchSize : 5000
}
<= div>
morphlines : [
{ id : 1=C2=A0
=C2=A0= commands : [
=C2=A0 { readLine { charset : UTF-8 } }
= =C2=A0 {
=C2=A0 =C2=A0 grok {
=C2=A0 =C2=A0 =C2=A0dicti= onaryFiles : [conf/grok-patterns]
=C2=A0 =C2=A0 =C2=A0expressions= : {
=C2=A0 =C2=A0 =C2=A0 =C2=A0message : """%{COM= BINEDAPACHELOG}"""
=C2=A0 =C2=A0 =C2=A0}
=C2=A0 =C2=A0 }
=C2=A0 }
# =C2=A0{ generateUUID {
# =C2=A0 =C2=A0 =C2=A0field : id
# =C2=A0 =C2=A0}
# =C2=A0}
=C2=A0 {
=C2=A0 =C2=A0 loadSolr {
= =C2=A0 =C2=A0 =C2=A0 solrLocator : ${SOLR_LOCATOR}
=C2=A0 =C2=A0 = }
=C2=A0 }
=C2=A0]
}
]
<= div>
I also have a grok-patterns file there, but I don't = think it's relevant.

Any ideas are much apprec= iated!

Thanks and best regards,
Radu
--
Performance Moni= toring * Log Analytics * Search Analytics
Solr & Elasticsearch Support *=C2= =A0http://sematext.com/




--001a11c3825a9df7b1051ff14af7--