nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Spark & NiFi question
Date Fri, 20 May 2016 13:50:35 GMT
I am very proud to be part of a community with threads like this!

On Fri, May 20, 2016 at 9:34 AM, Bryan Bende <bbende@gmail.com> wrote:
> Hi Conrad,
>
> Sorry this has been so challenging to setup. After trying it out myself, I
> believe the problem you ran into when you didn't set the System properties
> is actually a legit bug in the SiteToSiteClient...
> I wrote it up in this JIRA [1], but the short answer is that it never uses
> those properties to create an SSLContext and ends up trying to make a normal
> connection to the https end-point, and thus ends up failing.
>
> I made some quick code changes to work around the above issue, and
> eventually got it working using Storm, since I don't have spark streaming
> setup. Here is what I did...
>
> In conf/nifi.properties I set the following:
>
> # Site to Site properties
> nifi.remote.input.socket.host=
> nifi.remote.input.socket.port=8088
> nifi.remote.input.secure=true
>
> # web properties #
> nifi.web.war.directory=./lib
> nifi.web.http.host=
> nifi.web.http.port=8080
> nifi.web.https.host=
> nifi.web.https.port=8443
> nifi.web.jetty.working.directory=./work/jetty
> nifi.web.jetty.threads=200
>
> # security properties #
> nifi.sensitive.props.key=
> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
> nifi.sensitive.props.provider=BC
>
> nifi.security.keystore=/path/to/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ks.jks
> nifi.security.keystoreType=JKS
> nifi.security.keystorePasswd=localtest
> nifi.security.keyPasswd=localtest
> nifi.security.truststore=/path/to/nifi//nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ts.jks
> nifi.security.truststoreType=JKS
> nifi.security.truststorePasswd=localtest
>
>
> I started NiFi and used the unsecure url (http://localhost:8080/nifi)  to
> create a flow with GenerateFlowFile -> Output Port named "Data for Storm".
>
> There is an example Storm topology that is part of the code base [2], so I
> started with that, and modified the SiteToSiteClientConfig:
>
> final SiteToSiteClientConfig inputConfig = new SiteToSiteClient.Builder()
>         .url("https://localhost:8443/nifi")
>         .portName("Data for Storm")
>
> .keystoreFilename("/path/to/nifi//nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ks.jks")
>         .keystoreType(KeystoreType.JKS)
>         .keystorePass("localtest")
>
> .truststoreFilename("/path/to/nifi//nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ts.jks")
>         .truststoreType(KeystoreType.JKS)
>         .truststorePass("localtest")
>         .buildConfig();
>
> Now of course setting those properties only worked because of local changes
> I made, but after that I got a 401 Unauthorized when I ran the topology,
> which I think was where you were originally at.
>
> I went back into the unsecure url and checked the users section and didn't
> see anything, so I think I was incorrect that it automatically creates a
> pending account.
> I then put that localhost cert into my browser (I already had it as p12 from
> something else) and I went to https://localhost:8443/nifi and it prompted
> for the account request and I submitted it.
> Went back to the unsecure UI and approved the account with role NiFi, then
> went to the Output Port and gave access to the localhost user.
>
> After that it was working... I think since you were already at the point of
> getting the 401, if you can just get the account created for that
> certificate and the access controls on the ports, then it should probably
> work using the System properties as a work around for now, but not totally
> sure.
>
> Again, sorry for all the confusion, definitely planning to address the JIRA
> soon.
>
> -Bryan
>
> [1] https://issues.apache.org/jira/browse/NIFI-1907
> [2]
> https://github.com/apache/nifi/blob/e12a79ea929a222a93fd64bfc63382441e31060f/nifi-external/nifi-storm-spout/src/test/java/org/apache/nifi/storm/NiFiStormTopology.java
>
>
> On Fri, May 20, 2016 at 4:16 AM, Conrad Crampton
> <conrad.crampton@secdata.com> wrote:
>>
>> Thanks for the pointers Bryan, however wrt your first suggestion. I tried
>> without setting SSL properties on System properties and get an unable to
>> find ssl path error – this gets resolved by doing as I have done (but of
>> course this may be a red herring). I initially tried setting on site builder
>> but got the same error as below – it appears to make no difference as to
>> what is logged in the nifi-users.log if I include SSL props on site builder
>> or not, I get the same error viz:
>>
>> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180]
>> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
>> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
>> L=Maidstone, ST=Kent, C=GB>) GET
>> https://yarn-cm1.m.xxxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
>> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759]
>> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
>> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
>> L=Maidstone, ST=Kent, C=GB>) GET
>> https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
>> 2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180]
>> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to
>> verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data
>> Europe Ltd, L=Maidstone, ST=Kent, C=GB
>>
>> I am using self signed certs if that makes a difference (but these work
>> fine on across the cluster). I am not seeing my spark user appear in the
>> list of users to grant access.
>>
>> I have turned on debug for ssl to see if that is throwing up anything but
>> nothing appears obvious – here is the snipet that I would expect errors to
>> be shown from that log.
>>
>> ... no IV derived for this protocol
>> %% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
>> NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
>> NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
>> *** Finished
>> verify_data:  { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
>> ***
>> NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
>> *** Finished
>> verify_data:  { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
>> ***
>> NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
>> NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
>>
>> I don’t really know enough about certificates and how client java apps
>> would use them wrt to the host name/ ip address etc. of details is included
>> in them. The nifi-user.log is showing access from a specific IP address
>> which clearly doesn’t match the CN details in the cert. Just clutching at
>> straws here!
>>
>> Any other suggestions?
>>
>> Thanks
>> Conrad
>>
>> From: Bryan Bende <bbende@gmail.com>
>> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
>> Date: Thursday, 19 May 2016 at 17:08
>> To: "users@nifi.apache.org" <users@nifi.apache.org>
>> Subject: Re: Spark & NiFi question
>>
>> Hi Conrad,
>>
>> I think there are a couple of things at play here...
>>
>> One is that the SSL properties need to be set on the
>> SiteToSiteClientBuilder, rather than through system properties. There should
>> be methods to set the keystore and other values.
>>
>> In a secured NiFi instance, the certificate you are authenticating with
>> (the keystore used by the s2s client) would need to have an account in NiFi,
>> and would need to have access to the output port.
>> If you attempt to make a request with that cert, and then you go into the
>> NiFi UI as another user, you should be able to go into the accounts section
>> (top right) and approve the account for that certificate.
>>
>> Then if you stop your output port, right-click and Configure... and from
>> the Access Controls tab started typing the DN from your cert and add that
>> user to the Allowed Users list. Hit Apply and started the port again.
>>
>> We probably need to document this better, or write up an article about it
>> somewhere.
>>
>> Let us know if its still not working.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton
>> <conrad.crampton@secdata.com> wrote:
>>>
>>> Hi,
>>> Tried following a couple of blog posts about this [1], [2], but neither
>>> of these refer to using NiFi in clustered environment with SSL and I suspect
>>> this is where I am hitting problems (but don’t know where).
>>>
>>> The blogs state that using an output port (in the root process group I.e.
>>> on main canvas) which I have done and tried to connect thus..
>>>
>>> System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
>>> System.setProperty("javax.net.ssl.keyStorePassword", “*****");
>>> System.setProperty("javax.net.ssl.trustStore", “/cacerts.jks");
>>>
>>> SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
>>>         .url("https://yarn-cm1.mis-cds.local:9090/nifi")
>>>         .portName("Spark test out")
>>>         .buildConfig();
>>>
>>> SparkConf sparkConf = new
>>> SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
>>> JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new
>>> Duration(5000));
>>> JavaReceiverInputDStream<NiFiDataPacket> packetStream =
>>> jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
>>>
>>> JavaDStream text = packetStream.map(dataPacket -> new
>>> String(dataPacket.getContent(), StandardCharsets.UTF_8));
>>> text.print();
>>> jssc.start();
>>> jssc.awaitTermination();
>>>
>>> The error I am getting is
>>>
>>> 16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with
>>> delay 2000 ms: Failed to receive data from NiFi
>>> java.io.IOException: Server returned HTTP response code: 401 for URL:
>>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>> at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>>> at
>>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
>>> at
>>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
>>> at
>>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
>>> at
>>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
>>> at
>>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
>>> at
>>> org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
>>> at
>>> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
>>> at
>>> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.io.IOException: Server returned HTTP response code: 401
>>> for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>>> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
>>> at
>>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
>>> at
>>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
>>> ... 7 more
>>>
>>> Any pointers would be helpful in getting this working. I don’t know if I
>>> have to set up a remote process group with the output port (not sure how
>>> this works), or what. When I go to
>>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser, I
>>> get an access denied error.
>>> I have created keystore and signed by the RootCA used to sign all the
>>> self signed certs for the cluster.
>>>
>>> Running 0.6.1, 6 node cluster.
>>>
>>> Thanks
>>> Conrad
>>>
>>> [1[ -
>>> https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
>>> [2] -
>>> https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
>>>
>>>
>>> SecureData, combating cyber threats
>>>
>>> ________________________________
>>>
>>> The information contained in this message or any of its attachments may
>>> be privileged and confidential and intended for the exclusive use of the
>>> intended recipient. If you are not the intended recipient any disclosure,
>>> reproduction, distribution or other dissemination or use of this
>>> communications is strictly prohibited. The views expressed in this email are
>>> those of the individual and not necessarily of SecureData Europe Ltd. Any
>>> prices quoted are only valid if followed up by a formal written quote.
>>>
>>> SecureData Europe Limited. Registered in England & Wales 04365896.
>>> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
>>> Maidstone, Kent, ME16 9NT
>>
>>
>>
>>
>> ***This email originated outside SecureData***
>>
>> Click here to report this email as spam.
>
>

Mime
View raw message