Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0F272200CB3 for ; Mon, 26 Jun 2017 14:39:33 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0E09F160BDE; Mon, 26 Jun 2017 12:39:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 06E1F160BD9 for ; Mon, 26 Jun 2017 14:39:31 +0200 (CEST) Received: (qmail 57911 invoked by uid 500); 26 Jun 2017 12:39:31 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 57902 invoked by uid 99); 26 Jun 2017 12:39:31 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jun 2017 12:39:31 +0000 Received: from mail-pf0-f171.google.com (mail-pf0-f171.google.com [209.85.192.171]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id D51761A0372 for ; Mon, 26 Jun 2017 12:39:30 +0000 (UTC) Received: by mail-pf0-f171.google.com with SMTP id q86so1782pfl.3 for ; Mon, 26 Jun 2017 05:39:30 -0700 (PDT) X-Gm-Message-State: AKS2vOyuvHCyhMJU/XQWLlxjDzvG5XnJK/PB+zP/To26seUaGGWWQqHz /78PjikzC60pzD9cOXglrrndpY19Xw== X-Received: by 10.84.200.39 with SMTP id s36mr1600232pld.9.1498480769549; Mon, 26 Jun 2017 05:39:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.100.130.136 with HTTP; Mon, 26 Jun 2017 05:39:14 -0700 (PDT) In-Reply-To: <371f9384-9411-7be1-3937-e425ce58235b@apache.org> References: <1498428555533-13988.post@n4.nabble.com> <371f9384-9411-7be1-3937-e425ce58235b@apache.org> From: Stephan Ewen Date: Mon, 26 Jun 2017 14:39:14 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: MapR libraries shading issue To: Chesnay Schepler Cc: user Content-Type: multipart/alternative; boundary="94eb2c145ce4f8cb3f0552dc3d72" archived-at: Mon, 26 Jun 2017 12:39:33 -0000 --94eb2c145ce4f8cb3f0552dc3d72 Content-Type: text/plain; charset="UTF-8" Okay, just curious because the guy mentioned the behavior changes with removing the MapR dependencies. Maybe these dependencies change the trust-store or certificate-store providers? On Mon, Jun 26, 2017 at 2:35 PM, Chesnay Schepler wrote: > This looks more like a certification problem as described here: > https://github.com/square/okhttp/issues/2746 > > I don't think that shading could have anything to do with this. > > > On 26.06.2017 00:09, ani.desh1512 wrote: > >> I am trying to use Flink (1.3.0) with MapR(5.2.1). Accordingly, I built >> Flink >> for Mapr as follows with maven 3.1: >> >> /mvn clean install -DskipTests -Dscala.version=2.10.6 -Pvendor-repos,mapr >> -Dhadoop.version=2.7.0-mapr-1703 -Dzookeeper.version=3.4.5-mapr-1604/ >> >> I, then added /opt/mapr/lib/* to Flink classpath, added Datadog metrics >> entry to config and to test the config, started flink service via: >> /./bin/jobmanager.sh start local/. >> In the jobmanager logs, I see the following error: >> >> /ERROR org.apache.flink.runtime.metrics.MetricRegistry - >> Could >> not instantiate metrics reporter dghttp. Metrics might not be >> exposed/reported. >> java.lang.IllegalStateException: Failed contacting Datadog to validate >> API >> key >> at >> org.apache.flink.metrics.datadog.DatadogHttpClient.validateA >> piKey(DatadogHttpClient.java:73) >> at >> org.apache.flink.metrics.datadog.DatadogHttpClient.( >> DatadogHttpClient.java:61) >> at >> org.apache.flink.metrics.datadog.DatadogHttpReporter.open( >> DatadogHttpReporter.java:104) >> at >> org.apache.flink.runtime.metrics.MetricRegistry.(Metri >> cRegistry.java:129) >> at >> org.apache.flink.runtime.taskexecutor.TaskManagerServices.fr >> omConfiguration(TaskManagerServices.java:188) >> at >> org.apache.flink.runtime.taskmanager.TaskManager$.startTaskM >> anagerComponentsAndActor(TaskManager.scala:1921) >> at >> org.apache.flink.runtime.jobmanager.JobManager$.startJobMana >> gerActors(JobManager.scala:2322) >> at >> org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$ >> 1(JobManager.scala:2053) >> at >> org.apache.flink.runtime.jobmanager.JobManager$.runJobManage >> r(JobManager.scala:2052) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2. >> apply$mcV$sp(JobManager.scala:2139) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2. >> apply(JobManager.scala:2117) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2. >> apply(JobManager.scala:2117) >> at scala.util.Try$.apply(Try.scala:161) >> at >> org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindE >> xception(JobManager.scala:2172) >> at >> org.apache.flink.runtime.jobmanager.JobManager$.runJobManage >> r(JobManager.scala:2117) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anon$10. >> call(JobManager.scala:1992) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anon$10. >> call(JobManager.scala:1990) >> at >> org.apache.flink.runtime.security.HadoopSecurityContext$1. >> run(HadoopSecurityContext.java:43) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >> upInformation.java:1595) >> at >> org.apache.flink.runtime.security.HadoopSecurityContext.runS >> ecured(HadoopSecurityContext.java:40) >> at >> org.apache.flink.runtime.jobmanager.JobManager$.main(JobMana >> ger.scala:1990) >> at >> org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) >> Caused by: javax.net.ssl.SSLHandshakeException: >> sun.security.validator.ValidatorException: PKIX path building failed: >> sun.security.provider.certpath.SunCertPathBuilderException: unable to >> find >> valid certi >> fication path to requested target >> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) >> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) >> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) >> at >> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHa >> ndshaker.java:1514) >> at >> sun.security.ssl.ClientHandshaker.processMessage(ClientHands >> haker.java:216) >> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) >> at sun.security.ssl.Handshaker.process_record(Handshaker.java: >> 961) >> at >> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) >> at >> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSo >> cketImpl.java:1375) >> at >> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) >> at >> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) >> at >> org.apache.flink.shaded.okhttp3.internal.connection.RealConn >> ection.connectTls(RealConnection.java:268) >> at >> org.apache.flink.shaded.okhttp3.internal.connection.RealConn >> ection.establishProtocol(RealConnection.java:238) >> at >> org.apache.flink.shaded.okhttp3.internal.connection.RealConn >> ection.connect(RealConnection.java:149) >> at >> org.apache.flink.shaded.okhttp3.internal.connection.StreamAl >> location.findConnection(StreamAllocation.java:192) >> at >> org.apache.flink.shaded.okhttp3.internal.connection.StreamAl >> location.findHealthyConnection(StreamAllocation.java:121) >> at >> org.apache.flink.shaded.okhttp3.internal.connection.StreamAl >> location.newStream(StreamAllocation.java:100)/ >> >> This error disappears when I remove the mapr libs from the >> Flink_Classpath. >> I encounter a similar error (SSL handshake exception, PKIX path build >> failed) when I try to use aws-sdk(1.11.123) jar in my code and submit that >> code to flink. >> >> I think the shaded libs are causing this error. Am I right in assuming >> that? >> Is there any workaround for this? >> >> >> >> >> >> >> >> -- >> View this message in context: http://apache-flink-user-maili >> ng-list-archive.2336050.n4.nabble.com/MapR-libraries-shad >> ing-issue-tp13988.html >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive at Nabble.com. >> >> > --94eb2c145ce4f8cb3f0552dc3d72 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Okay, just curious because the guy mentioned the behavior = changes with removing the MapR dependencies.

Maybe these= dependencies change the trust-store or certificate-store providers?
<= div class=3D"gmail_extra">
On Mon, Jun 26, 20= 17 at 2:35 PM, Chesnay Schepler <chesnay@apache.org> wrote:=
This looks more like a certification pro= blem as described here: https://github.com/square/okhttp/issues/2746

I don't think that shading could have anything to do with this.


On 26.06.2017 00:09, ani.desh1512 wrote:
I am trying to use Flink (1.3.0) with MapR(5.2.1). Accordingly, I built Fli= nk
for Mapr as follows with maven 3.1:

/mvn clean install -DskipTests -Dscala.version=3D2.10.6 -Pvendor-repos,mapr=
-Dhadoop.version=3D2.7.0-mapr-1703 -Dzookeeper.version=3D3.4.5-mapr-1604/

I, then added /opt/mapr/lib/* to Flink classpath, added Datadog metrics
entry to config and to test the config, started flink service via:
/./bin/jobmanager.sh start local/.
In the jobmanager logs, I see the following error:

/ERROR org.apache.flink.runtime.metrics.MetricRegistry=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0- Could
not instantiate metrics reporter dghttp. Metrics might not be
exposed/reported.
java.lang.IllegalStateException: Failed contacting Datadog to validate= API
key
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.metrics.datadog.DatadogHttpClient.validateApiKey= (DatadogHttpClient.java:73)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.metrics.datadog.DatadogHttpClient.<init>(D= atadogHttpClient.java:61)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.metrics.datadog.DatadogHttpReporter.open(Datadog= HttpReporter.java:104)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.metrics.MetricRegistry.<init>(MetricRegistry.java:129)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.taskexecutor.TaskManagerServices.fromConfiguration(TaskManage= rServices.java:188)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.taskmanager.TaskManager$.startTaskManage= rComponentsAndActor(TaskManager.scala:1921)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerAc= tors(JobManager.scala:2322)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$1(Job= Manager.scala:2053)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(Job= Manager.scala:2052)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$m= cV$sp(JobManager.scala:2139)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(J= obManager.scala:2117)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(J= obManager.scala:2117)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at scala.util.Try$.apply(Try.scala:1= 61)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindExcept= ion(JobManager.scala:2172)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(Job= Manager.scala:2117)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobM= anager.scala:1992)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobM= anager.scala:1990)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(Had= oopSecurityContext.java:43)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.security.AccessController.do= Privileged(Native Method)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.java:422)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInf= ormation.java:1595)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecure= d(HadoopSecurityContext.java:40)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.s= cala:1990)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.sc= ala)
Caused by: javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:<= br> sun.security.provider.certpath.SunCertPathBuilderException: unable to = find
valid certi
fication path to requested target
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.security.ssl.Alerts.getSSLExc= eption(Alerts.java:192)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.security.ssl.SSLSocketImpl.fa= tal(SSLSocketImpl.java:1949)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.security.ssl.Handshaker.fatal= SE(Handshaker.java:302)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.security.ssl.Handshaker.fatal= SE(Handshaker.java:296)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandsha= ker.java:1514)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker= .java:216)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.security.ssl.Handshaker.processLoop(Handshaker.jav= a:1026)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.security.ssl.Handshaker.process_record(Handshaker.= java:961)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketI= mpl.java:1375)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:= 1403)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:= 1387)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.shaded.okhttp3.internal.connection.RealConnectio= n.connectTls(RealConnection.java:268)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.shaded.okhttp3.internal.connection.RealConnectio= n.establishProtocol(RealConnection.java:238)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.shaded.okhttp3.internal.connection.RealConnectio= n.connect(RealConnection.java:149)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.shaded.okhttp3.internal.connection.StreamAllocat= ion.findConnection(StreamAllocation.java:192)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.shaded.okhttp3.internal.connection.StreamAllocat= ion.findHealthyConnection(StreamAllocation.java:121)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.flink.shaded.okhttp3.internal.connection.StreamAllocat= ion.newStream(StreamAllocation.java:100)/

This error disappears when I remove the mapr libs from the Flink_Classpath.=
I encounter a similar error (SSL handshake exception, PKIX path build
failed) when I try to use aws-sdk(1.11.123) jar in my code and submit that<= br> code to flink.

I think the shaded libs are causing this error. Am I right in assuming that= ?
Is there any workaround for this?







--
View this message in context: http://apache-flink-user-mailin= g-list-archive.2336050.n4.nabble.com/MapR-libraries-shading-issue= -tp13988.html
Sent from the Apache Flink User Mailing List archive. mailing list archive = at Nabble.com.



--94eb2c145ce4f8cb3f0552dc3d72--