Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id DD294200C2A for ; Wed, 1 Mar 2017 15:14:23 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id DBAD6160B70; Wed, 1 Mar 2017 14:14:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3E850160B5E for ; Wed, 1 Mar 2017 15:14:22 +0100 (CET) Received: (qmail 84981 invoked by uid 500); 1 Mar 2017 14:14:16 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 84972 invoked by uid 99); 1 Mar 2017 14:14:16 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2017 14:14:16 +0000 Received: from MacBook.mail (1-164-151-173.dynamic.hinet.net [1.164.151.173]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 213E61A0046 for ; Wed, 1 Mar 2017 14:14:14 +0000 (UTC) Date: Wed, 1 Mar 2017 22:14:12 +0800 From: "Tzu-Li (Gordon) Tai" To: user@flink.apache.org Message-ID: In-Reply-To: References: Subject: Re: unclear exception when writing to elasticsearch X-Mailer: Airmail (397) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="58b6d734_3f8ebd22_a408" archived-at: Wed, 01 Mar 2017 14:14:24 -0000 --58b6d734_3f8ebd22_a408 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi Martin, I followed your setup: 1. Maven java quick start archetype (=46link version 1.1.3) 2. Added =60flink-connector-elasticsearch2=5F2.10=60 version 1.1.3 depend= ency 3. Ran the example in the =46link Elasticsearch docs against a Elasticsea= rch 2.4.1 installation and everything worked fine. Just to make sure nothing is conflicting, you could also try to do a =60m= vn dependency:purge-local-repository=60 on your project, and then re-down= load the dependencies with =60mvn clean install=60, and finally re-import= ing your project in the IDE. Let me know if this works for you=21 Cheers, Gordon On March 1, 2017 at 9:23:35 PM, Tzu-Li (Gordon) Tai (tzulitai=40apache.or= g) wrote: Hi Martin, Just letting you know I=E2=80=99m trying your setup right now, and will g= et back to you once I confirm the results. - Gordon On March 1, 2017 at 9:15:16 PM, Martin Neumann (mneumann=40sics.se) wrote= : I created the project using the maven archetype=C2=A0so I'm using the pac= kaged version pulled by maven.=C2=A0 At this point, I just try to run it directly from inside the IDE (Intelli= J), mostly since I don't want to build it and deploy it on the cluster al= l the time. I tried building it (maven 3.0.5), it builds fine but fails t= o run on the cluster with the same exception that I get if I run things f= rom within the IDE.=C2=A0 My guess is that maybe some function names have changed between elastic s= earch versions and they are just not compatible anymore. In the Worst case, I will hack something together that just writes the da= ta using=C2=A0HttpURLConnection pushing things to the rest interface. (If= that works from within flink) cheers Martin On Wed, Mar 1, 2017 at 12:24 PM, =46lavio Pompermaier wrote: Did you build =46link from sources or are you using the packeged version=3F= =C2=A0 Because I had an annoying problem when compiling =46link with maven > 3.3= . =46rom https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/= building.html=23dependency-shading: Maven 3.0.x, 3.1.x, and 3.2.x=C2=A0It is sufficient to call=C2=A0mvn clea= n install -DskipTests=C2=A0in the root directory of =46link code base. Maven 3.3.x=C2=A0The build has to be done in two steps: =46irst in the ba= se directory, then in the distribution project: mvn clean install -DskipTests cd flink-dist mvn clean install Note:=C2=A0To check your Maven version, run=C2=A0mvn --version.=C2=A0 On Wed, Mar 1, 2017 at 12:19 PM, Martin Neumann wrot= e: I tried to change the elastic search version to 2.4.1 which results in a = new exception: Caused by: java.lang.NoSuchMethodError: com.google.common.util.concurrent= .MoreExecutors.directExecutor()Ljava/util/concurrent/Executor; at org.elasticsearch.threadpool.ThreadPool.(ThreadPool.java:192) at org.elasticsearch.client.transport.TransportClient=24Builder.build(Tra= nsportClient.java:131) at org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink= .open(ElasticsearchSink.java:164) at org.apache.flink.api.common.functions.util.=46unctionUtils.open=46unct= ion(=46unctionUtils.java:38) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.ope= n(AbstractUdfStreamOperator.java:91) at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(S= treamTask.java:376) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.= java:256) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585) at java.lang.Thread.run(Thread.java:745) On Wed, Mar 1, 2017 at 7:58 AM, Tzu-Li (Gordon) Tai wrote: Hi Martin, You can do that by adding a dependency to the Elasticsearch client of you= r desired version in your project. You can also check what Elasticsearch client version the project is using= by checking =60mvn dependency:tree=60 from the base directory of your pr= oject. Cheers, Gordon On March 1, 2017 at 1:21:56 AM, Martin Neumann (mneumann=40sics.se) wrote= : Hej, thanks for the fast reply. I'm currently running things from inside my IDE so it should not be a pac= kaging problem. That said I added the plugin from the link provided but I= 'm not sure what elastic search library is needed. Where do I override the elastic search version=3F The only thing I'm curr= ently using is the flink-connector do I have to modify its code=3F org.apache.flink flink-connector-elasticsearch2=5F2.10 1.1.3 One thing I forgot to mention, I can only modify things locally packing i= t into a jar. I'm stuck with stock =46link 1.1.3 for the execution since = I'm running things on top of Hopsworks. cheers Martin On Tue, =46eb 28, 2017 at 5:42 PM, Tzu-Li (Gordon) Tai wrote: Hi=21 This could be a Elasticsearch server / client version conflict, or that t= he uber jar of your code wasn=E2=80=99t built properly. =46or the first possible issue, we=E2=80=99re currently using Elasticsear= ch 2.3.5 to build the =46link Elasticsearch Connector. Could you try over= riding this version to 2.4.1 when building your code and see if the probl= em remains=3F =46or the second issue, please check out=C2=A0https://ci.apache.org/proje= cts/flink/flink-docs-release-1.3/dev/linking.html=23packaging-dependencie= s-with-your-usercode-with-maven. Let me know if the problem remains after trying out the above :-) Cheers, Gordon On March 1, 2017 at 12:24:08 AM, Martin Neumann (mneumann=40sics.se) wrot= e: Hej, I'm trying to write to elastic search from a streaming application and I = get a weird error message I that I can't decipher. Hopefully, someone her= e can help me. I'm trying to run the=C2=A0java example=C2=A0from the webs= ite.I doublechecked that I can reach the elastic search from the developm= ent machine by putting some data in with curl. Has anyone an idea what th= e problem is=3F=C2=A0 Technical info: =46link 1.1.3 Elasticsearch 2.4.1 http://bbc2.sics.se:19208/ =7B =22name=22 : =22hopsworks=22, =22cluster=5Fname=22 : =22hops=22, =22cluster=5Fuuid=22 : =22XIVrGHeaTc2nICQC85chpw=22, =22version=22 : =7B =22number=22 : =222.4.1=22, =22build=5Fhash=22 : =22c67dc32e24162035d18d6fe1e952c4cbcbe79d16=22, =22build=5Ftimestamp=22 : =222016-09-27T18:57:55Z=22, =22build=5Fsnapshot=22 : false, =22lucene=5Fversion=22 : =225.5.2=22 =7D, =22tagline=22 : =22You Know, for Search=22=C2=A0 =7D=C2=A0 Changes in the code: Map config =3D new HashMap<>(); // This instructs the sink to emit after every element, otherwise they wo= uld be buffered config.put(=22bulk.flush.max.actions=22, =221=22); config.put(=22cluster.name=22, =22hops=22); ArrayList transports =3D new ArrayList<>(); transports.add(new InetSocketAddress(InetAddress.getByName(=22bbc2.sics.s= e=22), 19208)); =C2=A0 Exception: Caused by: java.lang.NoClassDef=46oundError: Could not initialize class o= rg.elasticsearch.threadpool.ThreadPool at org.elasticsearch.client.transport.TransportClient=24Builder.build(Tra= nsportClient.java:133) at org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink= .open(ElasticsearchSink.java:164) at org.apache.flink.api.common.functions.util.=46unctionUtils.open=46unct= ion(=46unctionUtils.java:38) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.ope= n(AbstractUdfStreamOperator.java:91) at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(S= treamTask.java:376) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.= java:256) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585) at java.lang.Thread.run(Thread.java:745) --58b6d734_3f8ebd22_a408 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline