flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniket Deshpande <ani.desh1...@gmail.com>
Subject Re: Classloader error after SSL setup
Date Wed, 04 Oct 2017 20:58:08 GMT
Hi Chesnay,
Thanks for the reply. After your suggestion, I found out that setting
*blob.service.ssl.enabled:
false* solved the issue and now all the pipelines run as expected.
So, the issue is kinda narrowed down to blob service ssl now.
I also checked the jobmanager logs when blob ssl is enabled and I see the
following error:










*2017-10-03 23:28:50.459 [BLOB connection for /<jm_ip>:46932] ERROR
org.apache.flink.runtime.blob.BlobServerConnection  - Error while executing
BLOB connection. javax.net.ssl.SSLHandshakeException: Received fatal alert:
certificate_unknown         at
sun.security.ssl.Alerts.getSSLException(Alerts.java:192)         at
sun.security.ssl.Alerts.getSSLException(Alerts.java:154)         at
sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
  at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
      at
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:71)         at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:119)
*
So, is there some additional steps that I have to follow for enabling SSL
for blob service?

On Wed, Oct 4, 2017 at 4:09 PM, Eron Wright <eronwright@gmail.com> wrote:

> By following Chesney's recommendation we will hopefully uncover an SSL
> error that is being masked.  Another thing to try is to disable hostname
> verification (it is enabled by default) to see whether the certificate is
> being rejected.
>
> On Wed, Oct 4, 2017 at 5:15 AM, Chesnay Schepler <chesnay@apache.org>
> wrote:
>
>> something that would also help us narrow down the problematic area is to
>> enable SSL for one component at a time and see
>> which one causesd the job to fail.
>>
>>
>> On 04.10.2017 14:11, Chesnay Schepler wrote:
>>
>> The configuration looks reasonable. Just to be sure, are the paths
>> accessible by all nodes?
>>
>> As a first step, could you set the logging level to DEBUG (by modifying
>> the 'conf/log4j.properties' file), resubmit the job (after a cluster
>> restart) and check the Job- and TaskManager logs for any exception?
>>
>> On 04.10.2017 03:15, Aniket Deshpande wrote:
>>
>> Background: We have a setup of Flink 1.3.1 along with a secure MAPR
>> cluster (Flink is running on mapr client nodes). We run this flink cluster
>> via flink-jobmanager.sh foreground and flink-taskmanager.sh foreground command
>> via Marathon.  In order for us to make this work, we had to add
>> -Djavax.net.ssl.trustStore="$JAVA_HOME/jre/lib/security/cacerts" in
>> flink-console.sh as extra JVM arg (otherwise, flink was taking MAPR's
>> ssl_truststore as default truststore and then we were facing issues for any
>> 3rd party jars like aws_sdk etc.). This entire setup was working fine as it
>> is and we could submit our jars and the pipelines ran without any problem
>>
>>
>> Problem: We started experimenting with enabling ssl for all
>> communication for Flink. For this, we followed https://ci.apache.org
>> /projects/flink/flink-docs-release-1.3/setup/security-ssl.html for
>> generating CA and keystore. I added the following properties to
>> flink-conf.yaml:
>>
>>
>> security.ssl.enabled: true
>> security.ssl.keystore: /opt/flink/certs/node1.keystore
>> security.ssl.keystore-password: <password>
>> security.ssl.key-password: <password>
>> security.ssl.truststore: /opt/flink/certs/ca.truststore
>> security.ssl.truststore-password: <password>
>> jobmanager.web.ssl.enabled: true
>> taskmanager.data.ssl.enabled: true
>> blob.service.ssl.enabled: true
>> akka.ssl.enabled: true
>>
>>
>> We then spin up a cluster and tried submitting the same job which was
>> working before. We get the following erros:
>> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
>> load user class: org.apache.flink.streaming.con
>> nectors.kafka.FlinkKafkaConsumer09
>> ClassLoader info: URL ClassLoader:
>> Class not resolvable through given classloader.
>>         at org.apache.flink.streaming.api.graph.StreamConfig.getStreamO
>> perator(StreamConfig.java:229)
>>         at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init
>> >(OperatorChain.java:95)
>>         at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(
>> StreamTask.java:230)
>>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>>
>> This error disappears when we remove the ssl config properties i.e run
>> flink cluster without ssl enabled.
>>
>>
>> So, did we miss any steps for enabling ssl?
>>
>>
>> P.S.: We tried removing the extra JVm arg mentioned above, but still get
>> the same error.
>>
>> --
>>
>> Aniket
>>
>>
>>
>>
>


-- 
Yours Sincerely,
Aniket S Deshpande.

Mime
View raw message