flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arif,Mubaraka" <arif.mubar...@heb.com>
Subject RE: Flume - elasticsearch sink fails
Date Mon, 02 May 2016 21:27:41 GMT
Hello Iain,

They are in my plugins.d dir of flume. Do I also need to have them in flume lib folder ?

Thanks,
Muby

From: iain wright [mailto:iainwrig@gmail.com]
Sent: Monday, May 02, 2016 4:25 PM
To: user@flume.apache.org
Subject: Re: Flume - elasticsearch sink fails

Hi Muby,

Are the ES and lucene-core jars in your flume lib folder?


--
Iain Wright

This email message is confidential, intended only for the recipient(s) named above and may
contain information that is privileged, exempt from disclosure under applicable law. If you
are not the intended recipient, do not disclose or disseminate the message to anyone except
the intended recipient. If you have received this message in error, or are not the named recipient(s),
please immediately notify the sender by return email, and delete all copies of this message.

On Mon, May 2, 2016 at 1:47 PM, Arif,Mubaraka <arif.mubaraka@heb.com<mailto:arif.mubaraka@heb.com>>
wrote:
Downgraded to Elasticsearch 1.7.5 and still get the same error :

RROR

org.apache.flume.lifecycle.LifecycleSupervisor

Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@730d4fcf counterGroup:{
name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
        at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
        at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
        at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
        at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
        at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
        at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


From: iain wright [mailto:iainwrig@gmail.com<mailto:iainwrig@gmail.com>]
Sent: Monday, May 02, 2016 2:52 PM
To: user@flume.apache.org<mailto:user@flume.apache.org>
Cc: Mabrito,Greg
Subject: Re: Flume - elasticsearch sink fails

Hi,

These look relevant to your goals:

https://github.com/elastic/elasticsearch/issues/14187<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elastic_elasticsearch_issues_14187&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=p6q-rN0ma3I-5GVFu0e_CASXMGrDeforSajvFopvNic&e=>

https://github.com/lucidfrontier45/ElasticsearchSink2<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lucidfrontier45_ElasticsearchSink2&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=RSyBce-WtursnyucSVeewjWRYtSDC1dkIbXOpE3boRc&e=>

HTH,

--
Iain Wright

This email message is confidential, intended only for the recipient(s) named above and may
contain information that is privileged, exempt from disclosure under applicable law. If you
are not the intended recipient, do not disclose or disseminate the message to anyone except
the intended recipient. If you have received this message in error, or are not the named recipient(s),
please immediately notify the sender by return email, and delete all copies of this message.

On Mon, May 2, 2016 at 12:44 PM, Arif,Mubaraka <arif.mubaraka@heb.com<mailto:arif.mubaraka@heb.com>>
wrote:
Hello,

Trying to flume tweets from Tibco-JMS source into elastic search sink.
Running Elasticsearch on our edge nodes independent of Hadoop.
Running cloudera Hadoop 5.4.8

On running the error is  :

May 2, 2:15:14.047 PM

INFO

org.apache.flume.instrumentation.MonitoredCounterGroup

Component type: SINK, name: k1 started

May 2, 2:15:14.050 PM

WARN

org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient

[arcapl0021707:9200]

May 2, 2:15:14.052 PM

ERROR

org.apache.flume.lifecycle.LifecycleSupervisor

Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{
name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
        at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
        at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
        at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
        at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
        at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
        at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)



Flume configuration for ElasticSearch is :

#confgiure elastic search sink
test.sinks.k1.type = elasticsearch
test.sinks.k1.hostNames = 172.17.43.207:9200<https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.43.207-3A9200&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=Ofhp-XmxFUHyihMfLqzDo19Z1xzqkXNLV0T4NQnGy2Q&e=>
test.sinks.k1.indexName = TwitterComp_index
test.sinks.k1.indexType = TwitterComp_type
test.sinks.k1.clusterName = twittersearch
test.sinks.k1.batchSize = 500
test.sinks.k1.ttl = 5d
test.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer

Any help is greatly appreciated.

Thanks,
Muby


Mime
View raw message