flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From أنس الليثي <dev.fano...@gmail.com>
Subject Re: Flume MorphlineSolrSink
Date Wed, 18 May 2016 09:44:01 GMT
I am sorry Baris, I could not find an exact solution.

Just a little trick you can try,
The exception say *there is not method named skipBytes in
the org.apache.lucene.util.fst.**FST class*.
Try to get the latest jar of apache lucene from mvnrepository.com and place
it in the lib directory of flume.
Hope that work fir you.

Best regards

On 18 May 2016 at 11:55, Baris Akgun (Garanti Teknoloji) <
BarisAkgu@garanti.com.tr> wrote:

> Hi
>
>
>
>
>
> The version of solr in cdh is *solr-4.10.3-cdh5.4.0*
>
>
>
> And the files in flume lib directory;
>
>
>
>
> [gbbdap36]/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib$ls
>
> activation-1.1.jar                                 hive-metastore.jar
>
> ant-1.8.1.jar                                      hive-serde.jar
>
> ant-launcher-1.8.1.jar                             hive-shims-0.23.jar
>
> antlr-2.7.7.jar                                    hive-shims-common.jar
>
> antlr-runtime-3.4.jar                              hive-shims.jar
>
> aopalliance-1.0.jar                                hive-shims-scheduler.jar
>
> apacheds-i18n-2.0.0-M15.jar                        htrace-core-3.0.4.jar
>
> apacheds-kerberos-codec-2.0.0-M15.jar              httpclient-4.2.5.jar
>
> apache-log4j-extras-1.1.jar                        httpcore-4.2.5.jar
>
> api-asn1-api-1.0.0-M20.jar                         irclib-1.10.jar
>
> api-util-1.0.0-M20.jar
> jackson-annotations-2.3.0.jar
>
> asm-3.2.jar                                        jackson-core-2.3.1.jar
>
> async-1.4.0.jar
>       jackson-core-asl-1.8.8.jar
>
> asynchbase-1.5.0.jar
> jackson-databind-2.3.1.jar
>
> avro-ipc.jar                                       jackson-jaxrs-1.8.8.jar
>
> avro.jar
> jackson-mapper-asl-1.8.8.jar
>
> bonecp-0.7.1.RELEASE.jar                           jackson-xc-1.8.8.jar
>
> calcite-avatica-1.0.0-incubating.jar               janino-2.7.6.jar
>
> calcite-core-1.0.0-incubating.jar
> jasper-compiler-5.5.23.jar
>
> calcite-linq4j-1.0.0-incubating.jar
> jasper-runtime-5.5.23.jar
>
> commons-beanutils-1.7.0.jar                        javax.inject-1.jar
>
> commons-beanutils-core-1.8.0.jar                   jaxb-api-2.2.2.jar
>
> commons-cli-1.2.jar                                jaxb-impl-2.2.3-1.jar
>
> commons-codec-1.8.jar                              jdo-api-3.0.1.jar
>
> commons-collections-3.2.1.jar                      jersey-client-1.9.jar
>
> commons-compiler-2.7.6.jar                         jersey-core-1.8.jar
>
> commons-compress-1.4.1.jar                         jersey-guice-1.9.jar
>
> commons-configuration-1.6.jar                      jersey-json-1.9.jar
>
> commons-dbcp-1.4.jar                               jersey-server-1.9.jar
>
> commons-digester-1.8.jar                           jersey-servlet-1.14.jar
>
> commons-el-1.0.jar                                 jets3t-0.6.1.jar
>
> commons-httpclient-3.1.jar                         jettison-1.1.jar
>
> commons-io-2.4.jar
> jetty-6.1.26.cloudera.4.jar
>
> commons-jexl-2.1.1.jar
>           jetty-util-6.1.26.cloudera.4.jar
>
> commons-lang-2.6.jar                               jline-2.11.jar
>
> commons-logging-1.1.3.jar                          joda-time-2.1.jar
>
> commons-math3-3.1.1.jar                            jopt-simple-3.2.jar
>
> commons-net-3.1.jar                                jsch-0.1.42.jar
>
> commons-pool-1.5.4.jar                             jsp-api-2.1.jar
>
> curator-client-2.6.0.jar                           jsr305-1.3.9.jar
>
> curator-framework-2.6.0.jar                        jta-1.1.jar
>
> curator-recipes-2.6.0.jar                          kafka_2.10-0.8.1.1.jar
>
> datanucleus-api-jdo-3.2.1.jar                      kite-data-core.jar
>
> datanucleus-core-3.2.2.jar                         kite-data-hbase.jar
>
> datanucleus-rdbms-3.2.1.jar                        kite-data-hive.jar
>
> derby-10.8.2.2.jar
> kite-hadoop-compatibility.jar
>
> eigenbase-properties-1.1.4.jar                     leveldbjni-all-1.8.jar
>
> flume-avro-source-1.5.0-cdh5.4.0.jar               libfb303-0.9.2.jar
>
> flume-dataset-sink-1.5.0-cdh5.4.0.jar              libthrift-0.9.2.jar
>
> flume-file-channel-1.5.0-cdh5.4.0.jar              log4j-1.2.17.jar
>
> flume-hdfs-sink-1.5.0-cdh5.4.0.jar                 logredactor-1.0.2.jar
>
> flume-hive-sink-1.5.0-cdh5.4.0.jar                 mapdb-0.9.9.jar
>
> flume-irc-sink-1.5.0-cdh5.4.0.jar                  metrics-core-2.2.0.jar
>
> flume-jdbc-channel-1.5.0-cdh5.4.0.jar              mina-core-2.0.4.jar
>
> flume-jms-source-1.5.0-cdh5.4.0.jar                netty-3.6.2.Final.jar
>
> flume-kafka-channel-1.5.0-cdh5.4.0.jar             opencsv-2.3.jar
>
> flume-kafka-source-1.5.0-cdh5.4.0.jar              paranamer-2.3.jar
>
> flume-ng-auth-1.5.0-cdh5.4.0.jar                   parquet-avro.jar
>
> flume-ng-configuration-1.5.0-cdh5.4.0.jar          parquet-column.jar
>
> flume-ng-core-1.5.0-cdh5.4.0.jar                   parquet-common.jar
>
> flume-ng-elasticsearch-sink-1.5.0-cdh5.4.0.jar     parquet-encoding.jar
>
> flume-ng-embedded-agent-1.5.0-cdh5.4.0.jar         parquet-format.jar
>
> flume-ng-hbase-sink-1.5.0-cdh5.4.0.jar             parquet-generator.jar
>
> flume-ng-kafka-sink-1.5.0-cdh5.4.0.jar
> parquet-hadoop-bundle.jar
>
> flume-ng-log4jappender-1.5.0-cdh5.4.0.jar          parquet-hadoop.jar
>
> flume-ng-morphline-solr-sink-1.5.0-cdh5.4.0.jar    parquet-jackson.jar
>
> flume-ng-node-1.5.0-cdh5.4.0.jar
> pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
>
> flume-ng-sdk-1.5.0-cdh5.4.0.jar                    protobuf-java-2.5.0.jar
>
> flume-scribe-source-1.5.0-cdh5.4.0.jar             scala-library-2.10.1.jar
>
> flume-spillable-memory-channel-1.5.0-cdh5.4.0.jar  serializer-2.7.1.jar
>
> flume-thrift-source-1.5.0-cdh5.4.0.jar
> servlet-api-2.5-20110124.jar
>
> flume-tools-1.5.0-cdh5.4.0.jar                     servlet-api-2.5.jar
>
> flume-twitter-source-1.5.0-cdh5.4.0.jar            slf4j-api-1.7.5.jar
>
> groovy-all-2.1.6.jar                               slf4j-log4j12-1.7.5.jar
>
> gson-2.2.4.jar                                     snappy-java-1.0.4.1.jar
>
> guava-11.0.2.jar                                   ST4-4.0.4.jar
>
> guice-3.0.jar                                      stax-api-1.0.1.jar
>
> guice-servlet-3.0.jar                              stax-api-1.0-2.jar
>
> hadoop-annotations.jar                             stringtemplate-3.2.1.jar
>
> hadoop-auth.jar                                    twitter4j-core-3.0.3.jar
>
> hadoop-common.jar
> twitter4j-media-support-3.0.3.jar
>
> hadoop-yarn-api.jar
> twitter4j-stream-3.0.3.jar
>
> hadoop-yarn-common.jar                             velocity-1.7.jar
>
> hadoop-yarn-server-applicationhistoryservice.jar   xalan-2.7.1.jar
>
> hadoop-yarn-server-common.jar                      xercesImpl-2.9.1.jar
>
> hadoop-yarn-server-resourcemanager.jar             xml-apis-1.3.04.jar
>
> hadoop-yarn-server-web-proxy.jar                   xmlenc-0.52.jar
>
> hive-ant.jar                                       xz-1.0.jar
>
> hive-common.jar                                    zkclient-0.3.jar
>
> hive-exec.jar
>
>
>
>
>
> Thanks
>
>
>
> *From:* أنس الليثي [mailto:dev.fanooos@gmail.com]
> *Sent:* Wednesday, May 18, 2016 11:28 AM
> *To:* user@flume.apache.org
> *Subject:* Re: Flume MorphlineSolrSink
>
>
>
> I am not sure the issue you shared is the same issue you are facing.
>
>
>
> *NoClassDefFoundError* is totally different than *NoSuchMethodError*.
>
>
>
> I think solving this problem without upgrading is easy.
>
>
>
> Could you please tell what is the version of Solr you installed? and also
> put a list of the files in the flume lib directory.
>
>
>
> Best regards
>
>
>
> On 18 May 2016 at 11:11, Baris Akgun (Garanti Teknoloji) <
> BarisAkgu@garanti.com.tr> wrote:
>
> Hi
>
>
>
> You re right. I also find some jira for that problem. (
> https://issues.apache.org/jira/browse/FLUME-2392) The problem is solved
> in 1.6.0 version.
>
>
>
> Unfortunately, we are using oracle bda with cdh 5.4 ,therefore it is not
> easy to upgrade any service (flume, hive etc.) . We depended the oracle for
> upgrading.
>
>
>
> How can we solve that problem without upgrading the flume? It must be some
> other solutions.
>
>
>
> Thanks..
>
>
>
> *From:* Gonzalo Herreros [mailto:gherreros@gmail.com]
> *Sent:* Wednesday, May 18, 2016 10:48 AM
> *To:* user
> *Subject:* Re: Flume MorphlineSolrSink
>
>
>
> Flume 1.5.0 is pretty old,
>
> Why don't you use version 1.6.0 included in CDH, that will ensure the
> library compatibility.
>
>
>
> On 18 May 2016 at 08:43, Baris Akgun (Garanti Teknoloji) <
> BarisAkgu@garanti.com.tr> wrote:
>
> Hi,
>
>
>
> I am trying to make real time indexing with using flume 1.5.0 and
> morphline solr sink. I am using packed base solr in my CDH.
>
>
>
>
>
> tier1.sinks = solrSink
>
> tier1.channels = channel1
>
> #
>
> tier1.channels.channel1.type=memory
>
> tier1.channels.channel1.capacity=1000000
>
> #
>
> tier1.sinks.solrSink.type =
> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>
> tier1.sinks.solrSink.batchSize = 100
>
> tier1.sinks.solrSink.batchDurationMillis = 10000
>
> tier1.sinks.solrSink.morphlineFile = morphlines.conf
>
> tier1.sinks.solrSink.morphlineId = morphline1
>
> #
>
> tier1.sinks.solrSink.channel = channel1
>
>
>
>
>
> The flume throw error in sink step.  The error is given below. What is
> your ideas to solve the problem?
>
>
>
> Thanks.
>
>
>
>
>
> ERROR
>
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@748016d1 counterGroup:{
> name:null counters:{} } } - Exception follows.
>
> java.lang.NoSuchMethodError:
> org.apache.lucene.util.fst.FST$BytesReader.skipBytes(J)V
>
>          at org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:1117)
>
>          at
> org.apache.lucene.util.fst.FST.readFirstRealTargetArc(FST.java:1030)
>
>          at org.apache.lucene.util.fst.FST.readRootArcs(FST.java:487)
>
>          at org.apache.lucene.util.fst.FST.cacheRootArcs(FST.java:475)
>
>          at org.apache.lucene.util.fst.FST.finish(FST.java:458)
>
>          at org.apache.lucene.util.fst.Builder.finish(Builder.java:464)
>
>          at
> org.apache.lucene.analysis.synonym.SynonymMap$Builder.build(SynonymMap.java:284)
>
>          at
> org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.loadSynonyms(FSTSynonymFilterFactory.java:145)
>
>          at
> org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.inform(FSTSynonymFilterFactory.java:112)
>
>          at
> org.apache.lucene.analysis.synonym.SynonymFilterFactory.inform(SynonymFilterFactory.java:89)
>
>          at
> org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:675)
>
>          at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:167)
>
>          at
> org.apache.solr.schema.IndexSchemaFactory.create(IndexSchemaFactory.java:55)
>
>          at
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:69)
>
>          at
> org.kitesdk.morphline.solr.SolrLocator.getIndexSchema(SolrLocator.java:181)
>
>          at
> org.kitesdk.morphline.solr.SanitizeUnknownSolrFieldsBuilder$SanitizeUnknownSolrFields.<init>(SanitizeUnknownSolrFieldsBuilder.java:70)
>
>          at
> org.kitesdk.morphline.solr.SanitizeUnknownSolrFieldsBuilder.build(SanitizeUnknownSolrFieldsBuilder.java:52)
>
>          at
> org.kitesdk.morphline.base.AbstractCommand.buildCommand(AbstractCommand.java:302)
>
>          at
> org.kitesdk.morphline.base.AbstractCommand.buildCommandChain(AbstractCommand.java:249)
>
>          at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:46)
>
>          at
> org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>
>          at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>
>          at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>
>          at
> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>
>          at
> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>
>          at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>
>          at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>
>          at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>
>          at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>          at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>          at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>
>
>
>
>
>
> *Barış Akgün*
> Analitik Veri Ambarı ve Büyük Veri Yönetimi
> Uzman
>
> Tel
>
> :
>
> Dahili
>
> :
>
> Faks
>
> :
>
>
>
> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir
> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis
> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune
> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde
> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da
> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin
> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,
> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve
> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>
> This message and attachments are confidential and intended solely for the
> individual(s) stated in this message. If you received this message although
> you are not the addressee, you are responsible to keep the message
> confidential. The sender has no responsibility for the accuracy or
> correctness of the information in the message and its attachments. Our
> company shall have no liability for any changes or late receiving, loss of
> integrity and confidentiality, viruses and any damages caused in anyway to
> your computer system.
>
>
>
> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir
> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis
> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune
> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde
> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da
> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin
> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,
> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve
> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>
> This message and attachments are confidential and intended solely for the
> individual(s) stated in this message. If you received this message although
> you are not the addressee, you are responsible to keep the message
> confidential. The sender has no responsibility for the accuracy or
> correctness of the information in the message and its attachments. Our
> company shall have no liability for any changes or late receiving, loss of
> integrity and confidentiality, viruses and any damages caused in anyway to
> your computer system.
>
>
>
>
>
> --
>
> Anas Rabei
> Senior Software Developer
> Mubasher.info
> anas.rabei@mubasher.info
> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir
> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis
> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune
> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde
> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da
> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin
> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,
> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve
> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>
> This message and attachments are confidential and intended solely for the
> individual(s) stated in this message. If you received this message although
> you are not the addressee, you are responsible to keep the message
> confidential. The sender has no responsibility for the accuracy or
> correctness of the information in the message and its attachments. Our
> company shall have no liability for any changes or late receiving, loss of
> integrity and confidentiality, viruses and any damages caused in anyway to
> your computer system.
>



-- 
Anas Rabei
Senior Software Developer
Mubasher.info
anas.rabei@mubasher.info

Mime
View raw message