nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [nifi] andrewmlim commented on a change in pull request #4193: NIFI-7319 Add walkthrough document
Date Sat, 11 Apr 2020 03:02:26 GMT
andrewmlim commented on a change in pull request #4193: NIFI-7319 Add walkthrough document
URL: https://github.com/apache/nifi/pull/4193#discussion_r407009008
 
 

 ##########
 File path: nifi-docs/src/main/asciidoc/walkthroughs.adoc
 ##########
 @@ -0,0 +1,1099 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+= Apache NiFi Walkthroughs
+Apache NiFi Team <dev@nifi.apache.org>
+:homepage: http://nifi.apache.org
+:linkattrs:
+
+== Purpose
+The intent of this document is to provide a canonical source of prescriptive instruction
sets for common administrator and user tasks using Apache NiFi. It is intended to complement
the link:overview.html[NiFi Overview^], link:administration-guide.html[NiFi System Administrator's
Guide^], and link:user-guide.html[NiFi User's Guide^]. Those documents provide extensive reference
information about various features and configuration values, but they do not provide information
on _how_ to accomplish tasks.
+
+This document is _not_ intended to be a comprehensive repository of all possible admin or
user activities, but rather a collection of well-documented reference activities which can
be followed directly or adapted to custom scenarios. This document requires continued updates
as application or ecosystem behavior changes, and corrections and improvements where the content
is unclear to the community. To contribute to this document, please link:https://issues.apache.org/jira/secure/CreateIssue!default.jspa[open
a Jira^] (see link:https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-WheretoStart?[further
instructions in the Contributor Guide^]) and link:https://github.com/apache/nifi/pulls[submit
a pull request (PR)^].
+
+NOTE: This document is provided with no warranty. All steps have been evaluated for correctness
to the extent possible by the Apache NiFi community, but no responsibility is assumed for
negative impacts on any computer system where these commands are executed.
+
+== Installing Apache NiFi
+
+Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data.
It supports powerful and scalable directed graphs of data routing, transformation, and system
mediation logic. In addition to NiFi, there is the NiFi Toolkit, a collection of command-line
tools which help perform administrative tasks such as interacting with remote services, generating
TLS certificates, managing nodes in a cluster, and encrypting sensitive configuration values.
 
+
+|=======================================================================================================================
+|Description        | Instructions on downloading the Apache NiFi application and Toolkit
+|Purpose            | To make the application available to run on the specified machine (e.g.
a local development environment)
+|Starting Point     | Machine running modern OS
+|Expected Outcome   | Latest version of Apache NiFi available to run on host with no configuration
+|Estimated Duration | 1 minute + download time
+|=======================================================================================================================
+
+NOTE: The following instructions are for installing a single node of NiFi. This guide assumes
Mac OS X / macOS 10.11.0+ but should work for any modern operating system. *nix commands are
used by default, but Windows equivalents are provided where available.
+
+. Go to the link:http://nifi.apache.org/download.html[Apache NiFi Downloads^] page.
+. Download the latest version of NiFi via the compressed binary files. For example, if the
latest version is 1.11.4:
+* `nifi-1.11.4-bin.tar.gz` [1.2 GB]
+* `nifi-toolkit-1.11.4-bin.tar.gz` [42 MB]
+. If you are directed to a mirror page, click the first link on the page to download the
respective archive file.
++
+image::install-download-link.png["Download site"]
+. Note that the `nifi-1.11.4-bin.tar.gz` and the `nifi-toolkit-1.11.4-bin.tar.gz` compressed
files are downloaded to your `Downloads` folder.
+. Download the GPG signature and checksums for those files. They are found on the initial
download page next to each binary file.
+* `gpg --verify -v nifi-1.11.4-bin.tar.gz.asc` -- Verifies the GPG signature provided on
the binary by the Release Manager (RM). See link:https://nifi.apache.org/gpg.html#verifying-a-release-signature[NiFi
GPG Guide: Verifying a Release Signature^] for further details
+* `shasum -a 256 nifi-1.11.4-bin.tar.gz` -- Calculates a SHA-256 checksum over the downloaded
artifact. This should be compared with the contents of `nifi-1.11.4-bin.tar.gz.sha256` for
equality
+* `shasum -a 512 nifi-1.11.4-bin.tar.gz` -- Calculates a SHA-512 checksum over the downloaded
artifact. This should be compared with the contents of `nifi-1.11.4-bin.tar.gz.sha512` for
equality
++
+image::verify-release-gpg-and-checksums.png["Verifying the GPG signature and checksums]
+. Extract the files from each `tar.gz` file. This can be done by double-clicking on the files
in the Finder or running the following commands in the terminal.
+  * `tar -xvzf nifi-1.11.4-bin.tar.gz` -- Uncompresses the `gzip` file and extracts the `tar`
archive contents to `nifi-1.11.4/`
+  * `tar -xvzf nifi-toolkit-1.11.4-bin.tar.gz` -- Uncompresses the `gzip` file and extracts
the `tar` archive contents to `nifi-toolkit-1.11.4/`
+. Optionally, move the folders to a more appropriate location.
+
+== Building NiFi From Source
+
+|=======================================================================================================================
+|Description        | Instructions on downloading the Apache NiFi source code and building
the application locally
+|Purpose            | To make the application (with potential code modifications) available
to run on the specified machine (e.g. a local development environment)
+|Starting Point     | Machine running modern OS
+|Expected Outcome   | Latest version of Apache NiFi available to run on host with no configuration
+|Estimated Duration | 10 - 35 minutes + download time
+|=======================================================================================================================
+
+NOTE: This guide assumes Mac OS X / macOS 10.11.0+ but should work for any modern operating
system. *nix commands are used by default, but Windows equivalents are provided where available.
+
+. Go to the link:http://nifi.apache.org/download.html[Apache NiFi Downloads^] page.
+. Download the latest version of NiFi source code via the compressed files. For example,
if the latest version is 1.11.4:
+* `nifi-1.11.4-source-release.zip` [53 MB]
+. If you are directed to a mirror page, click the first link on the page to download the
respective archive file.
+. Note that the `nifi-1.11.4-source-release.zip` compressed file is downloaded to your `Downloads`
folder.
+. Download the GPG signature and checksums for the file. They are found on the initial download
page next to the archive file.
+* `gpg --verify -v nifi-1.11.4-source-release.zip.asc` -- Verifies the GPG signature provided
on the archive by the Release Manager (RM). See link:https://nifi.apache.org/gpg.html#verifying-a-release-signature[NiFi
GPG Guide: Verifying a Release Signature^] for further details
+* `shasum -a 256 nifi-1.11.4-source-release.zip` -- Calculates a SHA-256 checksum over the
downloaded artifact. This should be compared with the contents of `nifi-1.11.4-source-release.zip.sha256`
for equality
+* `shasum -a 512 nifi-1.11.4-source-release.zip` -- Calculates a SHA-512 checksum over the
downloaded artifact. This should be compared with the contents of `nifi-1.11.4-source-release.zip.sha512`
for equality
+. Extract the files from the `zip` file. This can be done by double-clicking on the file
in the Finder or running the following command in the terminal.
+* `unzip nifi-1.11.4-source-release.zip` -- Uncompresses the `zip` file and extracts the
archive contents to `nifi-1.11.4/`
+. Optionally, move the folder to a more appropriate location.
++
+[source]
+----
+nifi-1.11.4/ % ls -alGh
+total 328
+drwxr-xr-x  25 alopresto  staff   800B Apr  6 15:48 ./
+drwxr-xr-x   8 alopresto  staff   256B Apr  4 18:01 ../
+drwxr-xr-x   4 alopresto  staff   128B Jan 22 15:10 .github/
+-rw-r--r--   1 alopresto  staff   254B Jan 22 15:10 DEPENDENCIES
+-rw-r--r--   1 alopresto  staff    66K Jan 22 15:10 KEYS
+-rw-r--r--   1 alopresto  staff    21K Jan 22 15:10 LICENSE
+-rw-r--r--   1 alopresto  staff   7.1K Jan 22 15:10 NOTICE
+-rw-r--r--   1 alopresto  staff   9.1K Jan 22 15:10 README.md
+-rw-r--r--   1 alopresto  staff   3.3K Jan 22 15:10 SECURITY.md
+drwxr-xr-x   5 alopresto  staff   160B Apr  6 15:44 nifi-api/
+drwxr-xr-x   8 alopresto  staff   256B Apr  6 15:50 nifi-assembly/
+drwxr-xr-x   5 alopresto  staff   160B Apr  6 15:45 nifi-bootstrap/
+drwxr-xr-x  23 alopresto  staff   736B Apr  6 15:44 nifi-commons/
+drwxr-xr-x   9 alopresto  staff   288B Apr  6 15:44 nifi-docker/
+drwxr-xr-x   7 alopresto  staff   224B Apr  6 15:44 nifi-docs/
+drwxr-xr-x   8 alopresto  staff   256B Apr  6 15:44 nifi-external/
+drwxr-xr-x   5 alopresto  staff   160B Apr  6 15:44 nifi-framework-api/
+drwxr-xr-x   6 alopresto  staff   192B Apr  6 15:44 nifi-maven-archetypes/
+drwxr-xr-x   5 alopresto  staff   160B Apr  6 15:45 nifi-mock/
+drwxr-xr-x  82 alopresto  staff   2.6K Apr  6 15:44 nifi-nar-bundles/
+drwxr-xr-x   7 alopresto  staff   224B Apr  6 15:44 nifi-system-tests/
+drwxr-xr-x  14 alopresto  staff   448B Apr  6 15:44 nifi-toolkit/
+-rw-r--r--   1 alopresto  staff    44K Jan 22 15:10 pom.xml
+----
+. Build the NiFi source using link:https://maven.apache.org/[Apache Maven^] from the source
root directory (`nifi-1.11.4/`) using one of the following commands. For more information,
see the link:https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide[NiFi Contributor
Guide^]. Estimated build times for each command on a modern professional laptop are listed
below; allow additional time for dependency library downloads on first build.
+* `mvn clean install -Pinclude-grpc` -- Builds the application (expected time ~30 minutes)
+* `mvn clean install -T2.0C` -- Builds the application with multiple parallel threads (expected
time ~15 minutes)
+* `mvn clean install -T2.0C -DskipTests` -- Builds the application with multiple parallel
threads and unit tests disabled (expected time ~6 minutes)
++
+[source]
+----
+nifi-1.11.4/ % mvn clean install -T2.0C -DskipTests
+...
+------------------------------------------------------------------------
+Reactor Summary for nifi 1.11.4:
+
+nifi ............................................... SUCCESS [  0.834 s]
+nifi-api ........................................... SUCCESS [  8.799 s]
+nifi-framework-api ................................. SUCCESS [ 12.020 s]
+nifi-commons ....................................... SUCCESS [  0.253 s]
+nifi-utils ......................................... SUCCESS [ 16.029 s]
+nifi-properties .................................... SUCCESS [  6.863 s]
+nifi-security-utils ................................ SUCCESS [ 14.682 s]
+nifi-nar-bundles ................................... SUCCESS [  0.279 s]
+nifi-framework-bundle .............................. SUCCESS [  0.358 s]
+nifi-framework ..................................... SUCCESS [  1.538 s]
+nifi-properties-loader ............................. SUCCESS [ 14.153 s]
+...
+nifi-system-test-suite ............................. SUCCESS [  3.999 s]
+------------------------------------------------------------------------
+BUILD SUCCESS
+------------------------------------------------------------------------
+Total time:  06:17 min (Wall Clock)
+Finished at: 2020-04-06T15:50:35-07:00
+------------------------------------------------------------------------
+----
+. The resulting application is available in the build directory (for 1.11.14: `nifi-assembly/target/nifi-1.11.4-bin/nifi-1.11.4`).
+* `cd nifi-assembly/target/nifi-1.11.4-bin/nifi-1.11.4` -- Changes directory to the application
root directory
++
+[source]
+----
+nifi-1.11.4/nifi-assembly/target/nifi-1.11.4-bin/nifi-1.11.4/ % ls -alGh
+drwxr-xr-x   10 alopresto  staff   320B Apr  6 15:50 ./
+drwxr-xr-x    3 alopresto  staff    96B Apr  6 15:50 ../
+-rw-r--r--    1 alopresto  staff   157K Jan 22 07:10 LICENSE
+-rw-r--r--    1 alopresto  staff    85K Jan 22 07:10 NOTICE
+-rw-r--r--    1 alopresto  staff   4.7K Jan 22 07:10 README
+drwxr-xr-x    8 alopresto  staff   256B Apr  6 15:50 bin/
+drwxr-xr-x   10 alopresto  staff   320B Apr  6 15:50 conf/
+drwxr-xr-x    3 alopresto  staff    96B Apr  6 15:50 docs/
+drwxr-xr-x    2 alopresto  staff    64B Jan 22 07:10 extensions/
+drwxr-xr-x  116 alopresto  staff   3.6K Apr  6 15:50 lib/
+----
+
+== Starting NiFi
+
+|=======================================================================================================================
+|Description        | Instructions on running the Apache NiFi application
+|Purpose            | To configure and run the application on the local machine
+|Starting Point     | <<installing-apache-nifi,Installing Apache NiFi>> or <<building-nifi-from-source,Building
NiFi from Source>>
+|Expected Outcome   | Latest version of Apache NiFi running on host with minimal configuration
+|Estimated Duration | 1 minute
+|=======================================================================================================================
+
+. Start in the directory where NiFi was downloaded and unarchived or built. For this section,
the directory `/etc/nifi-1.11.4` will be used. This directory will be referred to as the "NiFi
home directory" and can be set explicitly (this is not done automatically by NiFi nor is it
required).
+* `export NIFI_HOME="/etc/nifi-1.11.4/"` -- Sets an environment variable (`$NIFI_HOME`) which
references the installation directory (optional)
+* `cd /etc/nifi-1.11.4` -- Changes the terminal to the NiFi home directory
+* `ls -alGh` -- Lists the contents of the directory (optional)
++
+image::nifi-home-dir-listing.png["NiFi Home directory contents"]
+. At this point, NiFi can be started with default configuration values (available at link:http://localhost:8080/nifi[`http://localhost:8080/nifi`^]).
+* `./bin/nifi.sh start` -- Starts NiFi
+* `tail -f logs/nifi-app.log` -- Tails the application log which will indicate when the application
has started
++
+image::nifi-app-log-ui-available.png["NiFi application log listing available URLS"]
+. Navigate to the URL in a web browser (link:http://localhost:8080/nifi[`http://localhost:8080/nifi`^]
or any listed in the `nifi-app.log` output).
++
+image::nifi-application-running-browser.png["NiFi application running in browser"]
+
+=== Modification of Configuration Values
+Many web applications run on `:8080` by default, so this port may already be occupied on
the machine. This section will demonstrate changing the port used by NiFi. All of the following
commands are run from `$NIFI_HOME`.
+
+. Ensure the application is not running.
+* `ps -ef | grep nifi` -- Checks the running system for any processes referencing `nifi`
(optional)
+* `./bin/nifi.sh status` -- Determines if the specified instance is running (optional)
+* `./bin/nifi.sh stop` -- Stops the specified instance
+. Open the main configuration file for NiFi (`nifi.properties`). Any text editor is sufficient
for this (Sublime Text, Atom, vi, nano, etc.)
+* `$EDITOR conf/nifi.properties` -- Opens the `nifi.properties` file for editing
+. Change `nifi.web.http.port=8080` -> `nifi.web.http.port=7777` -- Sets the HTTP port
to a new value
++
+NOTE: Setting a port between 1 - 1024 requires `root` access on *nix systems
+. Save and close the file.
+. Start NiFi again
+* `./bin/nifi.sh start` -- Starts NiFi
+* `tail -f logs/nifi-app.log` -- Tails the application log
+. Open a web browser to the new address (link:http://localhost:7777/nifi[`http://localhost:7777/nifi`^])
+
+== Securing NiFi with TLS
+
+Apache NiFi requires link:https://en.wikipedia.org/wiki/Transport_Layer_Security[Transport
Layer Security (TLS)^] configuration for multiple reasons.
+
+. To encrypt communication between clients and server (this provides confidentialty and integrity
over transmitted data)
+. To prevent malicious users from intercepting data or impersonating the server
+. To enable any authentication & authorization mechanisms
+
+NiFi intentionally does not allow any authentication or authorization features over plaintext
HTTP. Without the confidentiality and integrity provided by TLS and the user & group access
controls, any malicious entity can intercept and modify NiFi API requests, corrupt and steal
data, and otherwise interfere with the NiFi instance. Because of NiFi's robust feature set,
this can even lead to complete control over the host running NiFi. For more information, see
link:administration-guide.html#security_configuration[Administrator's Guide: Security Configuration^].
+
+=== Securing NiFi with TLS Toolkit
+
+NOTE: This section assumes no enterprise IT department to provide signed certificates. For
a scenario with provided certificates, see <<securing-nifi-with-provided-certificates,Securing
NiFi with Provided Certificates>>.
+
+|=======================================================================================================================
+|Description        | Instructions on enabling link:https://en.wikipedia.org/wiki/Transport_Layer_Security[Transport
Layer Security (TLS)^] for the Apache NiFi application using the TLS Toolkit
+|Purpose            | Securing NiFi with TLS protects data in motion and is required to enable
authentication & authorization
+|Starting Point     | <<starting-nifi>>
+|Expected Outcome   | Latest version of Apache NiFi running on host over TLS with client
certificate authentication and authorization enabled and a single configured user
+|Estimated Duration | 10 minutes
+|=======================================================================================================================
+
+Apache NiFi provides a toolkit (a collection of command-line tools for system administration).
One of these is the TLS Toolkit, which provides a self-signed link:https://en.wikipedia.org/wiki/Certificate_authority[Certificate
Authority (CA)^] and can easily issue and sign certificates in the format expected by NiFi.
The toolkit can be run by a user or scripted to perform automated certificate generation.
For more information, see link:toolkit-guide.html#tls_toolkit[NiFi Toolkit Guide: TLS Toolkit^].
+
+The end result will consist of a self-signed NiFi CA (the root), a keystore and truststore
containing the necessary certificates for the NiFi instance to operate, and a client certificate
for a human user to access NiFi.
+
+image::nifi-tls-toolkit-standalone-cert-diagram.png["NiFi TLS Toolkit Standalone Certificate
Diagram"]
+
+. Start in the directory where the NiFi toolkit was downloaded and unarchived. For this section,
the directory `/etc/nifi-toolkit-1.11.4` will be used. This directory will be referred to
as the "NiFi Toolkit home directory" and can be set explicitly (this is not done automatically
by NiFi Toolkit nor is it required).
+* `export NIFI_TOOLKIT_HOME="/etc/nifi-toolkit-1.11.4/"` -- Sets an environment variable
(`$NIFI_TOOLKIT_HOME`) which references the installation directory (optional)
+* `cd /etc/nifi-toolkit-1.11.4` -- Changes the terminal to the NiFi Toolkit home directory
+* `ls -alGh` -- Lists the contents of the directory (optional)
++
+image::nifi-toolkit-home-dir-listing.png["NiFi Toolkit Home directory contents"]
+. Generate the certificate and key for the NiFi instance. Running this command will first
generate the NiFi CA root certificate and private key, then generate and sign the certificate
for the application instance, and finally generate a pre-configured `nifi.properties` file.
+* `./bin/tls-toolkit.sh standalone -n "localhost"` -- Generates the signed certificate for
`localhost`
++
+[source]
+----
+nifi-toolkit-1.11.4 % ./bin/tls-toolkit.sh standalone -n "localhost"
+2020/04/04 19:13:29 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine:
No nifiPropertiesFile specified, using embedded one.
+2020/04/04 19:13:29 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Running standalone certificate generation with output directory ../nifi-toolkit-1.11.4
+2020/04/04 19:13:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Generated new CA certificate ../nifi-toolkit-1.11.4/nifi-cert.pem and key ../nifi-toolkit-1.11.4/nifi-key.key
+2020/04/04 19:13:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Writing new ssl configuration to ../nifi-toolkit-1.11.4/localhost
+2020/04/04 19:13:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Successfully generated TLS configuration for localhost 1 in ../nifi-toolkit-1.11.4/localhost
+2020/04/04 19:13:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
No clientCertDn specified, not generating any client certificates.
+2020/04/04 19:13:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
tls-toolkit standalone completed successfully
+----
+. The toolkit has created three files in the `localhost` directory: `keystore.jks`, `truststore.jks`,
and `nifi.properties`. To see what was automatically populated in `nifi.properties`, compare
it to the default file in the NiFi instance.
+* `diff /etc/nifi-1.11.4/conf/nifi.properties localhost/nifi.properties` -- Compares the
original configuration with the newly-generated one
++
+--
+[source]
+----
+nifi-toolkit-1.11.4 % diff ../nifi-1.11.4/conf/nifi.properties localhost/nifi.properties
+135,137c135,137
+< nifi.remote.input.host=
+< nifi.remote.input.secure=false
+< nifi.remote.input.socket.port=
+---
+> nifi.remote.input.host=localhost
+> nifi.remote.input.secure=true
+> nifi.remote.input.socket.port=10443
+145c145
+< nifi.web.http.port=8080
+---
+> nifi.web.http.port=
+147,148c147,148
+< nifi.web.https.host=
+< nifi.web.https.port=
+---
+> nifi.web.https.host=localhost
+> nifi.web.https.port=9443
+163,169c163,169
+< nifi.security.keystore=
+< nifi.security.keystoreType=
+< nifi.security.keystorePasswd=
+< nifi.security.keyPasswd=
+< nifi.security.truststore=
+< nifi.security.truststoreType=
+< nifi.security.truststorePasswd=
+---
+> nifi.security.keystore=./conf/keystore.jks
+> nifi.security.keystoreType=jks
+> nifi.security.keystorePasswd=aCeVndQ8JxIi9kLoz9YS65RClHPxB516tmIA/n26b54
+> nifi.security.keyPasswd=aCeVndQ8JxIi9kLoz9YS65RClHPxB516tmIA/n26b54
+> nifi.security.truststore=./conf/truststore.jks
+> nifi.security.truststoreType=jks
+> nifi.security.truststorePasswd=hbuVighksEPIxl6iGl1WFCIrFqdb65KZuamj72J7Yp8
+213c213
+< nifi.cluster.protocol.is.secure=false
+---
+> nifi.cluster.protocol.is.secure=true
+217,218c217,218
+< nifi.cluster.node.address=
+< nifi.cluster.node.protocol.port=
+---
+> nifi.cluster.node.address=localhost
+> nifi.cluster.node.protocol.port=11443
+----
+
+The `nifi.remote` section configures link:user-guide.html#site-to-site[Site to Site^] connections.
The `nifi.web` section disables the plaintext HTTP connector and enables an HTTPS connector
at `localhost:9443` (the default HTTPS port for NiFi). The `nifi.security` section populates
the paths to the keystore and truststore and randomly-generated passwords for each. The `nifi.cluster`
section configures the link:administration-guide.html#clustering[cluster communication protocol^]
(not used in a standalone instance).
+--
+. Copy the contents of the `localhost` directory to the `conf/` directory in the location
of the NiFi installation. **This command will overwrite any existing `nifi.properties` file
or keystore/truststore present in the destination.**
+* `cp -rv ./localhost/* /etc/nifi-1.11.4/conf/.` -- Copies the generated files into the NiFi
instance
+. Generate the user's client certificate to authenticate to NiFi. The toolkit will create
the certificate and sign it with the same NiFi CA certificate used for the NiFi server certificate.
Because the NiFi truststore includes this public certificate, it will trust the client certificate
and allow it to authenticate.
+* `./bin/tls-toolkit.sh standalone -C "CN=my_username, OU=NiFi"` -- Generates and signs the
client certificate
++
+[source]
+----
+nifi-toolkit-1.11.4 % ./bin/tls-toolkit.sh standalone -C "CN=my_username, OU=NiFi"
+2020/04/04 19:38:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine:
No nifiPropertiesFile specified, using embedded one.
+2020/04/04 19:38:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Running standalone certificate generation with output directory ../nifi-toolkit-1.11.4
+2020/04/04 19:38:30 INFO [main] org.apache.nifi.toolkit.tls.util.TlsHelper: Verifying the
certificate signature for CN=localhost,OU=NIFI
+2020/04/04 19:38:30 INFO [main] org.apache.nifi.toolkit.tls.util.TlsHelper: Attempting to
verify certificate CN=localhost,OU=NIFI signature with CN=localhost,OU=NIFI
+2020/04/04 19:38:30 INFO [main] org.apache.nifi.toolkit.tls.util.TlsHelper: Certificate was
signed by CN=localhost,OU=NIFI
+2020/04/04 19:38:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Using existing CA certificate ../nifi-toolkit-1.11.4/nifi-cert.pem and key ../nifi-toolkit-1.11.4/nifi-key.key
+2020/04/04 19:38:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
No hostnames specified, not generating any host certificates or configuration.
+2020/04/04 19:38:30 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Generating new client certificate ../nifi-toolkit-1.11.4/CN=my_username_OU=NiFi.p12
+2020/04/04 19:38:31 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Successfully generated client certificate ../nifi-toolkit-1.11.4/CN=my_username_OU=NiFi.p12
+2020/04/04 19:38:31 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
tls-toolkit standalone completed successfully
+----
+
+[[load_client_certificate_to_browser]]
+[start=6]
+. Load the client certificate into the web browser. Some browsers (e.g. Mozilla Firefox)
maintain their own internal keychain separate from the operating system's (OS). Others (e.g.
Apple Safari, Google Chrome) rely on the OS keychain. On most modern OS, double-clicking the
PKCS12 keystore (`CN=my_username_OU=NiFi.p12`) will open it with the default handler and load
it into the OS keychain. The randomly-generated password is available in `CN=my_username_OU=NiFi.password`.
+. Populate the *Initial Admin Identity* in NiFi. This is the identity that will be allowed
to access the NiFi instance and configure user and group access via the UI or API.
+* `$EDITOR /etc/nifi-1.11.4/conf/authorizers.xml` -- Opens the `authorizers.xml` file in
the text editor.
+* Replace `<property name="Initial User Identity 1"></property>` in the `userGroupProvider`
section with `<property name="Initial User Identity 1">CN=my_username, OU=NiFi</property>`
containing the full Distinguished Name (DN) as provided to the TLS Toolkit in the previous
step
+* Replace `<property name="Initial Admin Identity"></property>` in the `accessPolicyProvider`
section with `<property name="Initial Admin Identity">CN=my_username, OU=NiFi</property>`
containing the full Distinguished Name (DN) as provided to the TLS Toolkit in the previous
step
++
+WARNING: This is a frequent problem area when configuring security for NiFi. The DN must
match *exactly* to be authenticated. Check capitalization and whitespace especially.
+. Start NiFi with the new configuration.
+* `cd /etc/nifi-1.11.4/` -- Changes to the NiFi home directory
+* `./bin/nifi.sh start` -- Starts NiFi
+* `tail -f logs/nifi-app.log` -- Tails the application log
++
+--
+Note that there are now log entries about the keystore and truststore being loaded and the
available URLs are HTTPS.
+[source]
+----
+2020-04-04 19:52:01,122 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@623ebac7{nifi-error,/,file:///Users/alopresto/Workspace/scratch/release_verification/nifi-1.11.4-bin/nifi-1.11.4/work/jetty/nifi-web-error-1.11.4.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.11.4.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.11.4.war}
+2020-04-04 19:52:01,155 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@b96fb73(nifi-key,h=[localhost],w=[])
for SslContextFactory@5edd911f[provider=null,keyStore=file:///Users/alopresto/Workspace/scratch/release_verification/nifi-1.11.4-bin/nifi-1.11.4/conf/keystore.jks,trustStore=file:///Users/alopresto/Workspace/scratch/release_verification/nifi-1.11.4-bin/nifi-1.11.4/conf/truststore.jks]
+2020-04-04 19:52:01,191 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@767191b1{SSL,[ssl,
http/1.1]}{localhost:9443}
+2020-04-04 19:52:01,192 INFO [main] org.eclipse.jetty.server.Server Started @24336ms
+2020-04-04 19:52:01,215 INFO [main] org.apache.nifi.nar.NarAutoLoader Starting NAR Auto-Loader
for directory ./extensions ...
+2020-04-04 19:52:01,216 INFO [main] org.apache.nifi.nar.NarAutoLoader NAR Auto-Loader started
+2020-04-04 19:52:01,216 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started.
The UI is available at the following URLs:
+2020-04-04 19:52:01,216 INFO [main] org.apache.nifi.web.server.JettyServer https://localhost:9443/nifi
+2020-04-04 19:52:01,218 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated
communication with Bootstrap
+2020-04-04 19:52:01,219 INFO [main] org.apache.nifi.NiFi Controller initialization took 16457023826
nanoseconds (16 seconds).
+----
+--
+. Open the web browser to link:https://localhost:9443/nifi[`https://localhost:9443/nifi`^].
The browser will present a warning that the site you are attempting to visit is insecure because
the NiFi certificate is not signed by a trusted CA. Make an exception and acknowledge the
risk (all communications are still encrypted even if the browser does not recognize the certificate).
The browser may prompt to select the client certificate you wish to present to NiFi. Choose
the entry for `CN=my_username` generated by the toolkit.
++
+--
+The browser warning of an insecure server certificate
+
+image::browser-warning-insecure-site.png["Browser warning on untrusted server TLS certificate"]
+
+The browser prompting for a client certificate to present
+
+image::browser-present-client-cert.png["Browser prompting for client certificate to present"]
+
+The application running. Note the user's DN in the top-right corner
+
+image::nifi-running-tls-client-certificate.png["NiFi running with user logged in by client
certificate]
+--
+
+==== Additional Notes
+
+* While it is recommended to configure NiFi with security immediately, some admins have already
run NiFi with other configuration modifications before securing it. The TLS Toolkit provides
an option to consume an existing `nifi.properties` file and make the security changes to it
rather than using a default template. Use `-f /etc/nifi-1.11.4/conf/nifi.properties` or `--nifiPropertiesFile
/etc/nifi-1.11.4/conf/nifi.properties` if this file already exists. For more toolkit options,
see link:toolkit-guide.html#tls_operation_modes[NiFi Toolkit Guide: TLS Toolkit Operation
Modes^].
+* To generate a certificate for a hostname other than `localhost`, use the `-n somehost.com`
argument. To run on the local machine, DNS settings must allow for this. Either modify the
`/etc/hosts` file or use a tool like link:https://en.wikipedia.org/wiki/Dnsmasq[dnsmasq^]
to set up custom DNS routing. This is beyond the scope of this document.
+* The browser will warn about an insecure connection because it does not trust the self-signed
CA certificate. To explicitly mark this certificate as trusted, follow the instructions for
the relevant OS and browser combination. For example, using macOS Catalina and Google Chrome:
++
+--
+. Open the NiFi CA cert in the Keychain Access app. Double-click on `nifi-cert.pem` or run
the following command:
+* `open /etc/nifi-toolkit-1.11.4/nifi-cert.pem` -- Opens the certificate in Keychain Access
+* Right-click the `localhost` certificate and select *Get Info*
+
+image::keychain-access-certificate-listing.png["Keychain Access listing certificates"]
+
+* Expand the *Trust* section in the dialog
+* Change *Secure Sockets Layer (SSL)* to *Always Trust*
+
+image::keychain-access-trust-certificate.png["Trusting the CA certificate for TLS/SSL"]
+
+* Close the dialog
+* Restart the browser
+* Navigate to link:https://localhost:9443/nifi[`https://localhost:9443/nifi`^]
+
+image::nifi-trusted-server-certificate.png["Browser showing trusted NiFi server certificate"]
+--
+
+=== Securing NiFi with Provided Certificates
+
+NOTE: This section assumes an enterprise IT department or other mechanism to provide signed
certificates. For a scenario with self-signed certificates, see <<securing-nifi-with-tls-toolkit,Securing
NiFi with TLS Toolkit>>.
+
+|=======================================================================================================================
+|Description        | Instructions on enabling link:https://en.wikipedia.org/wiki/Transport_Layer_Security[Transport
Layer Security (TLS)^] for the Apache NiFi application using provided certificates
+|Purpose            | Securing NiFi with TLS protects data in motion and is required to enable
authentication & authorization
+|Starting Point     | <<starting-nifi>>
+|Expected Outcome   | Latest version of Apache NiFi running on host over TLS with client
certificate authentication and authorization enabled and a single configured user
+|Estimated Duration | 10 minutes
+|=======================================================================================================================
+
+In this scenario, a commercial CA is used, but it could also be an enterprise CA with an
internal CA certificate which is configured to be trusted by corporate machines. There are
many commercial vendors providing signed certificates. For this example, link:https://tinycert.org[TinyCert^]
is used as it is free. link:https://letsencrypt.org[Let's Encrypt^] is another free provider,
but requires verification of hostname ownership which involves additional steps not shown
here.
+
+For this scenario, all certificates and keys will be in link:https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail[Privacy-Enhanced
Mail (PEM)^] format. This is a common format for certificates and keys. It uses Base64 encoding
for the contents of the files, and can be transmitted across various mediums and storage mechanisms
easily. Plain text editors can be used to view the contents, and the `openssl` and `keytool`
tools can parse the contents.
+
+Example PEM file:
+
+[source]
+----
+% cat nifi.pem
+-----BEGIN CERTIFICATE-----
+MIIFDzCCA/egAwIBAgICURswDQYJKoZIhvcNAQELBQAwgZMxCzAJBgNVBAYTAlVT
+MQswCQYDVQQIDAJDQTEVMBMGA1UEBwwMU2FudGEgTW9uaWNhMRcwFQYDVQQKDA5B
+cGFjaGUgTmlGaSBDQTErMCkGA1UECwwiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNh
+dGUgU2lnbmluZzEaMBgGA1UEAwwRQXBhY2hlIE5pRmkgQ0EgQ0EwHhcNMjAwNDA1
+MjI0MTQ5WhcNMjEwNDA1MjI0MTQ5WjCBgjELMAkGA1UEBhMCVVMxCzAJBgNVBAgM
+AkNBMRUwEwYDVQQHDAxTYW50YSBNb25pY2ExFzAVBgNVBAoMDkFwYWNoZSBOaUZp
+IENBMSAwHgYDVQQLDBdBcGFjaGUgTmlGaSBXYWxrdGhyb3VnaDEUMBIGA1UEAwwL
+c2VjdXJlLm5pZmkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDJySMX
+H+Fco7WbXIKQ1u1RrMd/FE7zl/69X/7Da6x4c6jlI8fy3MbxZjqFnDsJpNuIkPVv
+yHcGCm8Lkw70DbCTUkW60MlVM5r4CkWhVgOd1RD34QzhgkcKjg29uOuCYa+FM78q
+E5Qp64wbLpDXHpxmm4/Qv97RHdTqynqRzYs6g+VyCn14nXuqJp0533F1T2khtK4z
+BrIMZj6VpWyyCFmJjmrW37GbcRuxMbtbgj+4mzD0Eew6/96R9A7Wxlq0QMuRTz12
+ie9xSi/GyzQV2r9gRzxuIo8qshMIq2d/1pipIgWNj2LzEXXoEbfHy7Jwm78e1G/+
+PV/ULIx+QL4h9Ni/AgMBAAGjggF6MIIBdjAJBgNVHRMEAjAAMB0GA1UdDgQWBBTy
+gw/GBUrqMI80gYpdlh3NNfwrBzAfBgNVHSMEGDAWgBRkovbp8RqbD0BeDyPcrBYg
+a1rzdDARBglghkgBhvhCAQEEBAMCBPAwCwYDVR0PBAQDAgP4MDsGA1UdJQQ0MDIG
+CCsGAQUFBwMBBggrBgEFBQcDAgYIKwYBBQUHAwMGCCsGAQUFBwMEBggrBgEFBQcD
+CDBtBggrBgEFBQcBAQRhMF8wLAYIKwYBBQUHMAGGIGh0dHA6Ly9vY3NwLnRpbnlj
+ZXJ0Lm9yZy9jYS0yNDA1MC8GCCsGAQUFBzAChiNodHRwOi8vYWlhLnRpbnljZXJ0
+Lm9yZy9jYS0yNDA1LmNydDA0BgNVHR8ELTArMCmgJ6AlhiNodHRwOi8vY3JsLnRp
+bnljZXJ0Lm9yZy9jYS0yNDA1LmNybDAnBgNVHREEIDAeggtzZWN1cmUubmlmaYcE
+fwAAAYIJbG9jYWxob3N0MA0GCSqGSIb3DQEBCwUAA4IBAQDAICxyfgm2eBa8J3+s
+D2QpIQgOc8fgMYeqWwgk5rHbDk8IdkH9XloAuSzxi/weZt3PQQOdNHeeOCOXEWAf
+n0X1SMGFUvLForHgArGolt9uFofvh2sE2q3/wSBI6J2940dYwZOPAlf5m7fNpcbz
+WCJZGt7Pn/VWm3+uPZrMAj+GzsvR9NVMZwK/eAFM4OKNCeiLRPv1qLARYVqvLJFK
+t9dlCrHKyDLIaUbG2Lcw6Yt7SBU7nAnobuYqImRRXm/bE0xwb9X9fD8UzJfmryOT
+Fvz4hlntwk1fgvG1n4SrZgFNZpg1awN5tXbwiOdisTwslQ49C4QCH5iEWCM1HExL
+A5GR
+-----END CERTIFICATE-----
+----
+
+Example parsed contents:
+
+[source]
+----
+% openssl x509 -in nifi.pem -text -noout
+Certificate:
+    Data:
+        Version: 3 (0x2)
+        Serial Number: 20763 (0x511b)
+    Signature Algorithm: sha256WithRSAEncryption
+        Issuer: C=US, ST=CA, L=Santa Monica, O=Apache NiFi CA, OU=Secure Digital Certificate
Signing, CN=Apache NiFi CA CA
+        Validity
+            Not Before: Apr  5 22:41:49 2020 GMT
+            Not After : Apr  5 22:41:49 2021 GMT
+        Subject: C=US, ST=CA, L=Santa Monica, O=Apache NiFi CA, OU=Apache NiFi Walkthrough,
CN=secure.nifi
+        Subject Public Key Info:
+            Public Key Algorithm: rsaEncryption
+                Public-Key: (2048 bit)
+                Modulus:
+                    00:c9:c9:23:17:1f:e1:5c:a3:b5:9b:5c:82:90:d6:
+                    ed:51:ac:c7:7f:14:4e:f3:97:fe:bd:5f:fe:c3:6b:
+                    ac:78:73:a8:e5:23:c7:f2:dc:c6:f1:66:3a:85:9c:
+                    3b:09:a4:db:88:90:f5:6f:c8:77:06:0a:6f:0b:93:
+                    0e:f4:0d:b0:93:52:45:ba:d0:c9:55:33:9a:f8:0a:
+                    45:a1:56:03:9d:d5:10:f7:e1:0c:e1:82:47:0a:8e:
+                    0d:bd:b8:eb:82:61:af:85:33:bf:2a:13:94:29:eb:
+                    8c:1b:2e:90:d7:1e:9c:66:9b:8f:d0:bf:de:d1:1d:
+                    d4:ea:ca:7a:91:cd:8b:3a:83:e5:72:0a:7d:78:9d:
+                    7b:aa:26:9d:39:df:71:75:4f:69:21:b4:ae:33:06:
+                    b2:0c:66:3e:95:a5:6c:b2:08:59:89:8e:6a:d6:df:
+                    b1:9b:71:1b:b1:31:bb:5b:82:3f:b8:9b:30:f4:11:
+                    ec:3a:ff:de:91:f4:0e:d6:c6:5a:b4:40:cb:91:4f:
+                    3d:76:89:ef:71:4a:2f:c6:cb:34:15:da:bf:60:47:
+                    3c:6e:22:8f:2a:b2:13:08:ab:67:7f:d6:98:a9:22:
+                    05:8d:8f:62:f3:11:75:e8:11:b7:c7:cb:b2:70:9b:
+                    bf:1e:d4:6f:fe:3d:5f:d4:2c:8c:7e:40:be:21:f4:
+                    d8:bf
+                Exponent: 65537 (0x10001)
+        X509v3 extensions:
+            X509v3 Basic Constraints:
+                CA:FALSE
+            X509v3 Subject Key Identifier:
+                F2:83:0F:C6:05:4A:EA:30:8F:34:81:8A:5D:96:1D:CD:35:FC:2B:07
+            X509v3 Authority Key Identifier:
+                keyid:64:A2:F6:E9:F1:1A:9B:0F:40:5E:0F:23:DC:AC:16:20:6B:5A:F3:74
+
+            Netscape Cert Type:
+                SSL Client, SSL Server, S/MIME, Object Signing
+            X509v3 Key Usage:
+                Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment,
Key Agreement
+            X509v3 Extended Key Usage:
+                TLS Web Server Authentication, TLS Web Client Authentication, Code Signing,
E-mail Protection, Time Stamping
+            Authority Information Access:
+                OCSP - URI:http://ocsp.tinycert.org/ca-2405
+                CA Issuers - URI:http://aia.tinycert.org/ca-2405.crt
+
+            X509v3 CRL Distribution Points:
+
+                Full Name:
+                  URI:http://crl.tinycert.org/ca-2405.crl
+
+            X509v3 Subject Alternative Name:
+                DNS:secure.nifi, IP Address:127.0.0.1, DNS:localhost
+    Signature Algorithm: sha256WithRSAEncryption
+         c0:20:2c:72:7e:09:b6:78:16:bc:27:7f:ac:0f:64:29:21:08:
+         0e:73:c7:e0:31:87:aa:5b:08:24:e6:b1:db:0e:4f:08:76:41:
+         fd:5e:5a:00:b9:2c:f1:8b:fc:1e:66:dd:cf:41:03:9d:34:77:
+         9e:38:23:97:11:60:1f:9f:45:f5:48:c1:85:52:f2:c5:a2:b1:
+         e0:02:b1:a8:96:df:6e:16:87:ef:87:6b:04:da:ad:ff:c1:20:
+         48:e8:9d:bd:e3:47:58:c1:93:8f:02:57:f9:9b:b7:cd:a5:c6:
+         f3:58:22:59:1a:de:cf:9f:f5:56:9b:7f:ae:3d:9a:cc:02:3f:
+         86:ce:cb:d1:f4:d5:4c:67:02:bf:78:01:4c:e0:e2:8d:09:e8:
+         8b:44:fb:f5:a8:b0:11:61:5a:af:2c:91:4a:b7:d7:65:0a:b1:
+         ca:c8:32:c8:69:46:c6:d8:b7:30:e9:8b:7b:48:15:3b:9c:09:
+         e8:6e:e6:2a:22:64:51:5e:6f:db:13:4c:70:6f:d5:fd:7c:3f:
+         14:cc:97:e6:af:23:93:16:fc:f8:86:59:ed:c2:4d:5f:82:f1:
+         b5:9f:84:ab:66:01:4d:66:98:35:6b:03:79:b5:76:f0:88:e7:
+         62:b1:3c:2c:95:0e:3d:0b:84:02:1f:98:84:58:23:35:1c:4c:
+         4b:03:91:91
+----
+
+The prerequisites for the scenario (issued by the IT department):
+
+* A signed NiFi server certificate for the specified host (`secure.nifi` for this example)
in PEM format (`nifi.pem`)
+* The matching private key in PEM format (`nifi.key`)
+* A signed client certificate for the specified user (`CN=my_username, ...` for this example)
in PEM format (`client.pem`)
+* The matching private key in PEM format (`client.key`)
+* The CA certificate in PEM format (`cacert.pem`)
+
+NOTE: For more information on converting certificates between various forms, see link:toolkit-guide.html#additional_certificate_commands[Toolkit
Guide: Additional Certificate Commands^].
 
 Review comment:
   Can keep link in current browser tab.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message