flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kartik Vashishta <kartik.u...@gmail.com>
Subject Re: Flume - elasticsearch sink fails
Date Mon, 02 May 2016 22:36:20 GMT
I think this will NOT work, it is not supported on the most recent releases
of flume and ES

On Mon, May 2, 2016 at 4:27 PM, Arif,Mubaraka <arif.mubaraka@heb.com> wrote:

> Hello Iain,
>
>
>
> They are in my plugins.d dir of flume. Do I also need to have them in
> flume lib folder ?
>
>
>
> Thanks,
>
> Muby
>
>
>
> *From:* iain wright [mailto:iainwrig@gmail.com]
> *Sent:* Monday, May 02, 2016 4:25 PM
> *To:* user@flume.apache.org
>
> *Subject:* Re: Flume - elasticsearch sink fails
>
>
>
> Hi Muby,
>
>
>
> Are the ES and lucene-core jars in your flume lib folder?
>
>
>
>
> --
>
> Iain Wright
>
>
>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
>
> On Mon, May 2, 2016 at 1:47 PM, Arif,Mubaraka <arif.mubaraka@heb.com>
> wrote:
>
> Downgraded to Elasticsearch 1.7.5 and still get the same error :
>
>
>
> RROR
>
> org.apache.flume.lifecycle.LifecycleSupervisor
>
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@730d4fcf counterGroup:{
> name:null counters:{} } } - Exception follows.
>
> java.lang.NoSuchMethodError:
> org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
>
>         at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
>
>         at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
>
>         at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
>
>         at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>
>         at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>
>         at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
> *From:* iain wright [mailto:iainwrig@gmail.com]
> *Sent:* Monday, May 02, 2016 2:52 PM
> *To:* user@flume.apache.org
> *Cc:* Mabrito,Greg
> *Subject:* Re: Flume - elasticsearch sink fails
>
>
>
> Hi,
>
>
>
> These look relevant to your goals:
>
>
>
> https://github.com/elastic/elasticsearch/issues/14187
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elastic_elasticsearch_issues_14187&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=p6q-rN0ma3I-5GVFu0e_CASXMGrDeforSajvFopvNic&e=>
>
>
>
> https://github.com/lucidfrontier45/ElasticsearchSink2
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lucidfrontier45_ElasticsearchSink2&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=RSyBce-WtursnyucSVeewjWRYtSDC1dkIbXOpE3boRc&e=>
>
>
>
> HTH,
>
>
> --
>
> Iain Wright
>
>
>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
>
> On Mon, May 2, 2016 at 12:44 PM, Arif,Mubaraka <arif.mubaraka@heb.com>
> wrote:
>
> Hello,
>
>
>
> Trying to flume tweets from Tibco-JMS source into elastic search sink.
>
> Running Elasticsearch on our edge nodes independent of Hadoop.
>
> Running cloudera Hadoop 5.4.8
>
>
>
> On running the error is  :
>
>
>
> May 2, 2:15:14.047 PM
>
> INFO
>
> org.apache.flume.instrumentation.MonitoredCounterGroup
>
> Component type: SINK, name: k1 started
>
> May 2, 2:15:14.050 PM
>
> WARN
>
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient
>
> [arcapl0021707:9200]
>
> May 2, 2:15:14.052 PM
>
> ERROR
>
> org.apache.flume.lifecycle.LifecycleSupervisor
>
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{
> name:null counters:{} } } - Exception follows.
>
> java.lang.NoSuchMethodError:
> org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
>
>         at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
>
>         at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
>
>         at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
>
>         at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>
>         at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>
>         at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> Flume configuration for ElasticSearch is :
>
>
>
> #confgiure elastic search sink
>
> test.sinks.k1.type = elasticsearch
>
> test.sinks.k1.hostNames = 172.17.43.207:9200
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.43.207-3A9200&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=Ofhp-XmxFUHyihMfLqzDo19Z1xzqkXNLV0T4NQnGy2Q&e=>
>
> test.sinks.k1.indexName = TwitterComp_index
>
> test.sinks.k1.indexType = TwitterComp_type
>
> test.sinks.k1.clusterName = twittersearch
>
> test.sinks.k1.batchSize = 500
>
> test.sinks.k1.ttl = 5d
>
> test.sinks.k1.serializer =
> org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
>
>
>
> Any help is greatly appreciated.
>
>
>
> Thanks,
>
> *Muby*
>
>
>
>
>

Mime
View raw message