nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Conrad Crampton <conrad.cramp...@SecData.com>
Subject Spark & NiFi question
Date Thu, 19 May 2016 15:54:35 GMT
Hi,
Tried following a couple of blog posts about this [1], [2], but neither of these refer to
using NiFi in clustered environment with SSL and I suspect this is where I am hitting problems
(but don’t know where).

The blogs state that using an output port (in the root process group I.e. on main canvas)
which I have done and tried to connect thus..

System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
System.setProperty("javax.net.ssl.keyStorePassword", “*****");
System.setProperty("javax.net.ssl.trustStore", “/cacerts.jks");

SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
        .url("https://yarn-cm1.mis-cds.local:9090/nifi")
        .portName("Spark test out")
        .buildConfig();

SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(5000));
JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new NiFiReceiver(config,
StorageLevel.MEMORY_ONLY()));

JavaDStream text = packetStream.map(dataPacket -> new String(dataPacket.getContent(), StandardCharsets.UTF_8));
text.print();
jssc.start();
jssc.awaitTermination();

The error I am getting is

16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Failed
to receive data from NiFi
java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
at org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
at org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
at org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
... 7 more

Any pointers would be helpful in getting this working. I don’t know if I have to set up
a remote process group with the output port (not sure how this works), or what. When I go
to https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser, I get an access
denied error.
I have created keystore and signed by the RootCA used to sign all the self signed certs for
the cluster.

Running 0.6.1, 6 node cluster.

Thanks
Conrad

[1[ - https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
[2] - https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark


SecureData, combating cyber threats
______________________________________________________________________ 
The information contained in this message or any of its attachments may be privileged and
confidential and intended for the exclusive use of the intended recipient. If you are not
the intended recipient any disclosure, reproduction, distribution or other dissemination or
use of this communications is strictly prohibited. The views expressed in this email are those
of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only
valid if followed up by a formal written quote.

SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address:
SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
Mime
View raw message