nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Gresock <jgres...@gmail.com>
Subject Re: 1.11.3 trust store error
Date Wed, 04 Mar 2020 20:15:31 GMT
The nifi.security.keyPasswd was not filled in, so it looked like this
(which is the default configuration):

nifi.security.keyPasswd=

On Wed, Mar 4, 2020 at 11:36 AM Endre Kovacs
<andrewsmith87@protonmail.com.invalid> wrote:

> Hi Nathan,
>
> There is already a ticket about this:
> https://issues.apache.org/jira/browse/NIFI-7219
>
> Best regards,
> Endre
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Wednesday, March 4, 2020 5:25 PM, Nathan Gough <thenatog@gmail.com>
> wrote:
>
> > I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and
> > I'm working on a fix for this.
> >
> > Nathan
> >
> > On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough thenatog@gmail.com wrote:
> >
> > > Hi Joe,
> > > Just to confirm here - was the nifi.security.keyPasswd not defined at
> all
> > > in your nifi.properties? Did you have to add the property and give it
> the
> > > correct value? Or was it in the nifi.properties file but blank? Or
> were the
> > > keyPasswd and keystorePasswd different values?
> > > Thanks,
> > > Nathan
> > > On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock jgresock@gmail.com wrote:
> > >
> > > > Yep, setting the nifi.security.keyPasswd to the same as
> > > > nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre!
> > > > On Tue, Mar 3, 2020 at 2:01 PM Joe Witt joe.witt@gmail.com wrote:
> > > >
> > > > > relevant change I believe is here:
> > > >
> > > >
> https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775
> > > >
> > > > > and
> > > > > is from https://issues.apache.org/jira/browse/NIFI-6927
> > > > > It looks to me like this was fixing an improper naming/usage issue
> > > > > that
> > > > > has been present but if so we probably should have addressed not
> in this
> > > > > bug fix line. Will defer to Troy/Andy for more context and next
> steps
> > > > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt joe.witt@gmail.com wrote:
> > > > >
> > > > > > If accurate....We need to look into whether this was a mistake
> and
> > > > > > fix it
> > > > >
> > > > > > if so. And we need to reflect this in the migration guide
> > > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward ryan.ward2@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > > Endre - thanks that was it
> > > > > > > On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
> > > > > > > andrewsmith87@protonmail.com.invalid wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > One additional thing:
> > > > > > > > we encountered something strange as well:
> > > > > > > > on 1.11.2 clustered, kerberized: request replication
worked
> well.
> > > > > > > > on 1.11.3 clustered, kerberized: request replication
did not
> work,
> > > > > > > > unless
> > > > > > > > you specify, and set
> > > > > > > > nifi.security.keyPasswd
> > > > > > > > to the very same password as the
> > > > > > > > nifi.security.keystorePasswd
> > > > > > > > For us this resolved the issue.
> > > > > > > > Best regards,
> > > > > > > > Endre
> > > > > > > > Sent with ProtonMail Secure Email.
> > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > > > > > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward <
> > > > > > > > ryan.ward2@gmail.com>
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Joe - Did you resolve your issue? If so I
am wondering
> what
> > > > > > > > > the
> > > > > > > > > fix
> > > > >
> > > > > > > > was as I'm seeing the same error on my cluster.
> > > > > > > >
> > > > > > > > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs
<
> > > > > > > > > andrewsmith87@protonmail.com.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Joe,
> > > > > > > > > >
> > > > > > > > > > 1.  Have you tried connecting/debugging
with openssl?
> From one
> > > > > > > > > >     pod
> > > > > > > > > >     to
> > > > > > > > > >
> > > > >
> > > > > > > > the other:
> > > > > > > >
> > > > > > > > > >     (openssl s_client -debug -CAfile
> > > > > > > > > >
> > > > > > > >
> > > > > > > > ca-bundle-signing-node-certificates.crt -cert
> my-client-cert.crt
> > > > > > > > -connect
> > > > > > > > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
> > > > > > > >
> > > > > > > > > > 2.  certs can also be verified by:
> > > > > > > > > >     openssl verify -verbose -CAfile ca-bundle.crt
> > > > > > > > > >     my-client-cert.crt
> > > > > > > > > >
> > > > >
> > > > > > > > > > 3.  Can you check if no intermediary CAs
are missing
> from the
> > > > > > > > > >     nodes
> > > > > > > > > >
> > > > >
> > > > > > > > truststore?
> > > > > > > >
> > > > > > > > > > 4.  This exception is coming from inter-node
> communication
> > > > > > > > > >     (replication
> > > > > > > > > >     of request from one node to the other).
This means
> that it is
> > > > > > > > > >     unrelated
> > > > > > > > > >
> > > > > >
> > > > > > > to
> > > > > > >
> > > > > > > > external user's authentication by client certificate.
The
> question
> > > > > > > > is:
> > > > >
> > > > > > > is
> > > > > > >
> > > > > > > > your inter node communication secured by the trusted
root CA
> (that
> > > > > > > > you
> > > > >
> > > > > > > are
> > > > > > >
> > > > > > > > sure that the CA cert is present in the trust store)
or is it
> > > > > > > > secured
> > > > > > > > by
> > > > >
> > > > > > > > selfsigned CA (which's CA may be lacking from your
> truststore)?
> > > > > > > >
> > > > > > > > > > 5.  `nifi.security.needClientAuth` is not
part of NiFi
> > > > > > > > > >     properties
> > > > > > > > > >     any
> > > > > > > > > >
> > > > >
> > > > > > > > more. If SSL is turned on, and no
> > > > > > > > `nifi.security.user.login.identity.provider` is set,
then
> client
> > > > > > > > cert
> > > > >
> > > > > > > based
> > > > > > >
> > > > > > > > auth is the default. But supplying this property have
no
> > > > > > > > detrimental
> > > > >
> > > > > > > effect
> > > > > > >
> > > > > > > > anyhow.
> > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Endre
> > > > > > > > > >
> > > > > > > > > >     Sent with ProtonMail Secure Email.
> > > > > > > > > >
> > > > > > > > > >     ‐‐‐‐‐‐‐ Original Message
‐‐‐‐‐‐‐
> > > > > > > > > >     On Wednesday, February 26, 2020 6:22
PM, Joe Gresock
> > > > > > > > > >
> > > > > > > >
> > > > > > > > jgresock@gmail.com wrote:
> > > > > > > >
> > > > > > > > > > > Were there any changes with how the
trust store is
> used in
> > > > > > > > > > > 1.11.3? I
> > > > > >
> > > > > > > > had a
> > > > > > > >
> > > > > > > > > > > 1.11.0 deployment working with the
following settings,
> but
> > > > > > > > > > > when I
> > > > >
> > > > > > > > deployed
> > > > > > > >
> > > > > > > > > > > 1.11.3, the cluster can't seem to replicate
requests
> to itself:
> > > > > > > > > > > nifi.remote.input.host=<redacted>
> > > > > > > > > > > nifi.remote.input.secure=true
> > > > > > > > > > > nifi.remote.input.socket.port=32440
> > > > > > > > > > > nifi.remote.input.http.enabled=true
> > > > > > > > > > > nifi.cluster.protocol.is.secure=true
> > > > > > > > > > > nifi.cluster.is.node=true
> > > >
> > > >
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > > >
> > > > > > > > > > > nifi.cluster.node.protocol.port=6007
> > > >
> > > >
> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> > > >
> > > > > > > > > > > nifi.web.https.port=8443
> > > > > > > > > > > nifi.security.keystore=./conf/keystore.jks
> > > > > > > > > > > nifi.security.keystoreType=jks
> > > > > > > > > > > nifi.security.keystorePasswd=<password>
> > > > > > > > > > > nifi.security.keyPasswd=
> > > > > > > > > > > nifi.security.truststore=./conf/truststore.jks
> > > > > > > > > > > nifi.security.truststoreType=jks
> > > > > > > > > > > nifi.security.truststorePasswd=<password>
> > > > > > > > > > > nifi.security.needClientAuth=true
> > > > > > > > > > > A trusted client cert that worked against
the old
> cluster is
> > > > > > > > > > > getting
> > > > > >
> > > > > > > > the
> > > > > > > >
> > > > > > > > > > > same trust error (PKIX path building
failed). I've
> verified
> > > > > > > > > > > that
> > > > > > > > > > > the
> > > > >
> > > > > > > > > > > client cert was issued by an issuer
that is definitely
> in the
> > > > > > > > > > > ./conf/truststore.jks as a trustedCertEntry.
> > > > > > > > > > > 2020-02-26 17:11:09,573 WARN [Replicate
Request
> Thread-7]
> > > > > > > > > > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > > > > > > > > > > javax.net.ssl.SSLHandshakeException:
> > > > > > > > > > > sun.security.validator.ValidatorException:
PKIX path
> building
> > > > > > > > > > > failed:
> > > > > > > >
> > > > > > > > > > >
> sun.security.provider.certpath.SunCertPathBuilderException:
> > > > > > > > > > > unable
> > > > >
> > > > > > > to
> > > > > > >
> > > > > > > > find
> > > > > > > >
> > > > > > > > > > > valid certification path to r
> > > > > > > > > > > equested target
> > > > > > > > > > > at
> sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > > > > > > > > at
> > > > > > > > > > >
> sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> > > > >
> > > > > > > > > > > at
> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> > > > > > > > > > > at
> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> > > >
> > > > > > > > > > > at
> > > > > > > > > > >
> sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> > > > >
> > > > > > > > > > > at
> > > > > > > > > > >
> sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> > > > >
> > > > > > > > > > > at
> > > >
> > > > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> > > >
> > > > > > > > > > > at
> > > > >
> > > > > okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
> > > > >
> > > > > > > > > > > at okhttp3.RealCall.execute(RealCall.java:77)
> > > > > > > > > > > at
> > > >
> > > >
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > >
> > > > > > > > > > > at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > > > > > at
> > > >
> > > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > >
> > > > > > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > > > > Caused by: sun.security.validator.ValidatorException:
> PKIX path
> > > > > > > > > > > building
> > > > > > > > >
> > > > > > > > > > > failed:
> > > > > > > > > > >
> sun.security.provider.certpath.SunCertPathBuilderException:
> > > > > >
> > > > > > > > unable
> > > > > > > >
> > > > > > > > > > > to find valid certification path to
requested target
> > > > > > > > > > > at
> > > > >
> > > > >
> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > >
> > > > > > > > > > > at
> > > > > > > > > > >
> sun.security.validator.Validator.validate(Validator.java:262)
> > > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
> > > >
> > > > > > > > > > > ... 35 common frames omitted
> > > > > > > > > > > Caused by:
> > > > > > > > > > >
> sun.security.provider.certpath.SunCertPathBuilderException:
> > > > > > > >
> > > > > > > > > > > unable to find valid certification
path to requested
> target
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
> > > >
> > > > > > > > > > > at
> > > >
> > > >
> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
> > > >
> > > > > > > > > > > at
> > > >
> > > > java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
> > > >
> > > > > > > > > > > at
> > > > >
> > > > >
> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
> > > > >
> > > > > > > > > > > ... 41 common frames omitted
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Joe
> > > >
> > > > --
> > > > Be on your guard; stand firm in the faith; be courageous; be strong.
> Do
> > > > everything in love. -1 Corinthians 16:13-14
>
>
>

-- 
Be on your guard; stand firm in the faith; be courageous; be strong.  Do
everything in love.    -*1 Corinthians 16:13-14*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message