pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] sijie closed pull request #2286: [website] cherry-pick "Rework TLS docs to separate concerns" to new website
Date Thu, 02 Aug 2018 17:40:54 GMT
sijie closed pull request #2286: [website] cherry-pick "Rework TLS docs to separate concerns"
to new website
URL: https://github.com/apache/incubator-pulsar/pull/2286
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/site2/docs/admin-api-clusters.md b/site2/docs/admin-api-clusters.md
index 91e700e8f6..e5adbf268a 100644
--- a/site2/docs/admin-api-clusters.md
+++ b/site2/docs/admin-api-clusters.md
@@ -79,7 +79,7 @@ bin/pulsar initialize-cluster-metadata \
   --broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651/
 ```
 
-You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls.md)
in your instance.
+You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md)
in your instance.
 
 ### Get configuration
 
diff --git a/site2/docs/admin-api-permissions.md b/site2/docs/admin-api-permissions.md
index 8a7793e9b1..51a93f65ea 100644
--- a/site2/docs/admin-api-permissions.md
+++ b/site2/docs/admin-api-permissions.md
@@ -4,6 +4,111 @@ title: Managing permissions
 sidebar_label: Persmissions
 ---
 
-## Permissions resources
+Permissions in Pulsar are managed at the [namespace](reference-terminology.md#namespace)
level
+(that is, within [tenants](reference-terminology.md#tenant) and [clusters](reference-terminology.md#cluster)).
 
-{% include explanations/permissions.md %}
+## Grant permissions
+
+You can grant permissions to specific roles for lists of operations such as `produce` and
`consume`.
+
+### pulsar-admin
+
+Use the [`grant-permission`](reference-pulsar-admin.md#grant-permission) subcommand and specify
a namespace, actions using the `--actions` flag, and a role using the `--role` flag:
+
+```shell
+$ pulsar-admin namespaces grant-permission test-tenant/ns1 \
+  --actions produce,consume \
+  --role admin10
+```
+
+Wildcard authorization can be performed when `authorizationAllowWildcardsMatching` is set
to `true` in `broker.conf`.
+
+e.g.
+```shell
+$ pulsar-admin namespaces grant-permission test-tenant/ns1 \
+                        --actions produce,consume \
+                        --role 'my.role.*'
+```
+
+Then, roles `my.role.1`, `my.role.2`, `my.role.foo`, `my.role.bar`, etc. can produce and
consume.  
+
+```shell
+$ pulsar-admin namespaces grant-permission test-tenant/ns1 \
+                        --actions produce,consume \
+                        --role '*.role.my'
+```
+
+Then, roles `1.role.my`, `2.role.my`, `foo.role.my`, `bar.role.my`, etc. can produce and
consume.
+
+**Note**: A wildcard matching works at **the beginning or end of the role name only**.
+
+e.g.
+```shell
+$ pulsar-admin namespaces grant-permission test-tenant/ns1 \
+                        --actions produce,consume \
+                        --role 'my.*.role'
+```
+
+In this case, only the role `my.*.role` has permissions.  
+Roles `my.1.role`, `my.2.role`, `my.foo.role`, `my.bar.role`, etc. **cannot** produce and
consume.
+
+### REST API
+
+{@inject: endpoint|POST|/admin/v2/namespaces/:tenant/:namespace/permissions/:role|operation/grantPermissionOnNamespace}
+
+### Java
+
+```java
+admin.namespaces().grantPermissionOnNamespace(namespace, role, getAuthActions(actions));
+```
+
+## Get permissions
+
+You can see which permissions have been granted to which roles in a namespace.
+
+### pulsar-admin
+
+Use the [`permissions`](reference-pulsar-admin#permissions) subcommand and specify a namespace:
+
+```shell
+$ pulsar-admin namespaces permissions test-tenant/ns1
+{
+  "admin10": [
+    "produce",
+    "consume"
+  ]
+}   
+```
+
+### REST API
+
+{@inject: endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/permissions|operation/getPermissions}
+
+### Java
+
+```java
+admin.namespaces().getPermissions(namespace);
+```
+
+## Revoke permissions
+
+You can revoke permissions from specific roles, which means that those roles will no longer
have access to the specified namespace.
+
+### pulsar-admin
+
+Use the [`revoke-permission`](reference-pulsar-admin.md#revoke-permission) subcommand and
specify a namespace and a role using the `--role` flag:
+
+```shell
+$ pulsar-admin namespaces revoke-permission test-tenant/ns1 \
+  --role admin10
+```
+
+### REST API
+
+{@inject: endpoint|DELETE|/admin/v2/namespaces/:tenant/:namespace/permissions/:role|operation/revokePermissionsOnNamespace}
+
+### Java
+
+```java
+admin.namespaces().revokePermissionsOnNamespace(namespace, role);
+```
diff --git a/site2/docs/client-libraries-go.md b/site2/docs/client-libraries-go.md
index eeae027483..e7a901efed 100644
--- a/site2/docs/client-libraries-go.md
+++ b/site2/docs/client-libraries-go.md
@@ -48,7 +48,7 @@ A URL for a production Pulsar cluster may look something like this:
 pulsar://pulsar.us-west.example.com:6650
 ```
 
-If you're using [TLS](security-tls.md) authentication, the URL will look like something like
this:
+If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like
something like this:
 
 ```http
 pulsar+ssl://pulsar.us-west.example.com:6651
@@ -435,7 +435,7 @@ Parameter | Description
 
 ## TLS encryption and authentication
 
-In order to use [TLS encryption](security-tls.md), you'll need to configure your client to
do so:
+In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your
client to do so:
 
  * Use `pulsar+ssl` URL type
  * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar
broker
diff --git a/site2/docs/client-libraries-java.md b/site2/docs/client-libraries-java.md
index 137f05bf39..c342e886d9 100644
--- a/site2/docs/client-libraries-java.md
+++ b/site2/docs/client-libraries-java.md
@@ -63,7 +63,7 @@ A URL for a production Pulsar cluster may look something like this:
 pulsar://pulsar.us-west.example.com:6650
 ```
 
-If you're using [TLS](security-tls.md) authentication, the URL will look like something like
this:
+If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like
something like this:
 
 ```http
 pulsar+ssl://pulsar.us-west.example.com:6651
@@ -407,11 +407,11 @@ The following schema formats are currently available for Java:
 
 ## Authentication
 
-Pulsar currently supports two authentication schemes: [TLS](security-tls.md) and [Athenz](security-athenz.md).
The Pulsar Java client can be used with both.
+Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md)
and [Athenz](security-athenz.md). The Pulsar Java client can be used with both.
 
 ### TLS Authentication
 
-To use [TLS](security-tls.md), you need to set TLS to `true` using the `setUseTls` method,
point your Pulsar client to a TLS cert path, and provide paths to cert and key files.
+To use [TLS](security-tls-authentication.md), you need to set TLS to `true` using the `setUseTls`
method, point your Pulsar client to a TLS cert path, and provide paths to cert and key files.
 
 Here's an example configuration:
 
diff --git a/site2/docs/concepts-architecture-overview.md b/site2/docs/concepts-architecture-overview.md
index a58c06feb0..fe19fb3baa 100644
--- a/site2/docs/concepts-architecture-overview.md
+++ b/site2/docs/concepts-architecture-overview.md
@@ -131,7 +131,7 @@ $ bin/pulsar proxy \
 Some important things to know about the Pulsar proxy:
 
 * Connecting clients don't need to provide *any* specific configuration to use the Pulsar
proxy. You won't need to update the client configuration for existing applications beyond
updating the IP used for the service URL (for example if you're running a load balancer over
the Pulsar proxy).
-* [TLS encryption and authentication](security-tls.md) is supported by the Pulsar proxy
+* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md)
is supported by the Pulsar proxy
 
 ## Service discovery
 
diff --git a/site2/docs/deploy-bare-metal-multi-cluster.md b/site2/docs/deploy-bare-metal-multi-cluster.md
index 2b94aa79d8..270ebdea46 100644
--- a/site2/docs/deploy-bare-metal-multi-cluster.md
+++ b/site2/docs/deploy-bare-metal-multi-cluster.md
@@ -211,7 +211,7 @@ As you can see from the example above, the following needs to be specified:
 * The web service URL for the cluster
 * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker)
in the cluster
 
-If you're using [TLS](security-tls.md), you'll also need to specify a TLS web service URL
for the cluster as well as a TLS broker service URL for the brokers in the cluster.
+If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service
URL for the cluster as well as a TLS broker service URL for the brokers in the cluster.
 
 Make sure to run `initialize-cluster-metadata` for each cluster in your instance.
 
diff --git a/site2/docs/deploy-bare-metal.md b/site2/docs/deploy-bare-metal.md
index 0c9236b5b7..14d4c66fd2 100644
--- a/site2/docs/deploy-bare-metal.md
+++ b/site2/docs/deploy-bare-metal.md
@@ -130,9 +130,9 @@ Flag | Description
 `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string
only needs to include *one* machine in the ZooKeeper cluster.
 `--configuration-store` | The configuration store connection string for the entire instance.
As with the `--zookeeper` flag, this connection string only needs to include *one* machine
in the ZooKeeper cluster.
 `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be
a standard DNS name. The default port is 8080 (we don't recommend using a different port).
-`--web-service-url-tls` | If you're using [TLS](security-tls.md), you'll also need to specify
a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using
a different port).
+`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need
to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend
using a different port).
 `--broker-service-url` | A broker service URL enabling interaction with the brokers in the
cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar`
scheme instead. The default port is 6650 (we don't recommend using a different port).
-`--broker-service-url-tls` | If you're using [TLS](security-tls.md), you'll also need to
specify a TLS web service URL for the cluster as well as a TLS broker service URL for the
brokers in the cluster. The default port is 6651 (we don't recommend using a different port).
+`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also
need to specify a TLS web service URL for the cluster as well as a TLS broker service URL
for the brokers in the cluster. The default port is 6651 (we don't recommend using a different
port).
 
 ## Deploying a BookKeeper cluster
 
diff --git a/site2/docs/security-authorization.md b/site2/docs/security-authorization.md
index 433f35051f..cf5a1e025b 100644
--- a/site2/docs/security-authorization.md
+++ b/site2/docs/security-authorization.md
@@ -4,17 +4,54 @@ title: Authentication and authorization in Pulsar
 sidebar_label: Authorization and ACLs
 ---
 
+
 In Pulsar, the [authentication provider](security-overview.md#authentication-providers) is
charged with properly identifying clients and
-associating them with [role tokens](security-overview.md#role-tokens). *Authorization* is
the process that determines *what* clients are able to do.
+associating them with [role tokens](security-overview.md#role-tokens). If only authentication
is enabled, an authenticated role token will have the ability to access all resources in the
cluster. *Authorization* is the process that determines *what* clients are able to do.
+
+The role tokens with the most privileges are the *superusers*. The *superusers* can create
and destroy tenants, along with having full access to all tenant resources.
+
+When a [tenant](reference-terminology.md#tenant) is created by a superuser, that tenant is
assigned an admin role. A client with the admin role token can then create, modify and destroy
namespaces, and grant and revoke permissions to *other role tokens* on those namespaces.
+
+## Broker and Proxy Setup
+
+### Enabling Authorization and Assigning Superusers
+
+Authorization is enabled and superusers are assigned in the broker ([`conf/broker.conf`](reference-configuration.md#broker))
and proxy ([`conf/proxy.conf`](reference-configuration.md#proxy)) configuration files.
+
+```properties
+authorizationEnabled=true
+superUserRoles=my-super-user-1,my-super-user-2
+```
+
+> A full list of parameters available in the `conf/broker.conf` file,
+> as well as the default values for those parameters, can be found in [Broker Configuration](reference-configuration.md#broker)

+
+Typically, superuser roles are used for administrators and clients but also for broker-to-broker
authorization. When using [geo-replication](concepts-replication.md), every broker needs to
be able to publish to all the other clusters' topics.
+
+### Proxy Roles
+
+By default, the broker treats the connection between a proxy and the broker as a normal user
connection. The user is authenticated as the role configured in ```proxy.conf``` (see ["Enabling
TLS Authentication on Proxies"](security-tls-authentication#on-proxies)). However, this is
rarely the behaviour that the user desires when connecting to the cluster through a proxy.
The user expects to be able to interact with the cluster as the role for which they have authenticated
with the proxy.
+
+Pulsar uses *Proxy roles* to enable this. Proxy roles are specified in the broker configuration
file, [`conf/broker.conf`](reference-configuration.md#broker). If a client that is authenticated
with a broker is one of its ```proxyRoles```, all requests from that client must also carry
information about the role of the client that is authenticated with the proxy. If this information,
which we call the *original principal*, is missing, the client will not be able to access
anything.
 
-Authorization in Pulsar is managed at the [tenant](reference-terminology.md#tenant) level,
which means that you can have multiple authorization schemes active
-in a single Pulsar instance. You could, for example, create a `shopping` tenant that has
one set of [roles](security-overview.md#role-tokens)
-and applies to a shopping application used by your company, while an `inventory` tenant would
be used only by an inventory application.
+Both the *proxy role* and the *original principle* must be authorized to access a resource
for that resource to be accessible via the proxy. Administrators can take two approaches to
this.
 
-> When working with properties, you can specify which of your Pulsar clusters your property
is allowed to use.
-> This enables you to also have cluster-level authorization schemes.
+The more secure approach is to grant access to the proxy roles each time you grant access
to a resource. For example, if you have a proxy role ```proxy1```, when a tenant is created
by the superuser, ```proxy1``` should be specified as one of the admin roles. When a role
is granted permissions to produce or consume from a namespace, if that client wants to produce
or consume through a proxy, ```proxy1``` should also be granted the same permissions.
 
-## Creating a new tenant
+Another approach is to make the proxy role a superuser. This will allow the proxy to access
all resources. The client will still need to authenticate with the proxy, and all requests
made through the proxy will have their role downgraded to the *original principal* of the
authenticated client. However, if the proxy is compromised, a bad actor could get full access
to your cluster.
+
+Roles can be specified as proxy roles in [`conf/broker.conf`](reference-configuration.md#broker).
+
+```properties
+proxyRoles=my-proxy-role
+
+# if you want to allow superusers to use the proxy (see above)
+superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role
+```
+
+## Administering Tenants
+
+### Creating a new tenant
 
 A Pulsar [tenant](reference-terminology.md#tenant) is typically provisioned by Pulsar [instance](reference-terminology.md#instance)
administrators or by some kind of self-service portal.
 
@@ -36,27 +73,11 @@ The structure of topic names in Pulsar reflects the hierarchy between
tenants, c
 persistent://tenant/namespace/topic
 ```
 
-## Managing permissions
-
-{% include explanations/permissions.md %}
-
-## Superusers
-
-In Pulsar you can assign certain roles to be *superusers* of the system. A superuser is allowed
to perform all administrative tasks on all tenants and namespaces, as well as to publish and
subscribe to all topics.
-
-Superusers are configured in the broker configuration file in [`conf/broker.conf`](reference-configuration.md#broker)
configuration file, using the [`superUserRoles`](reference-configuration.md#broker-superUserRoles)
parameter:
-
-```tenants
-superUserRoles=my-super-user-1,my-super-user-2
-```
-
-> A full listing of parameters available in the `conf/broker.conf` file, as well as the
default
-> values for those parameters, can be found in [Broker Configuration](reference-configuration.md#broker).
+### Managing permissions
 
-Typically, superuser roles are used for administrators and clients but also for broker-to-broker
authorization. When using [geo-replication](administration-geo.md), every broker
-needs to be able to publish to other clusters' topics.
+You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar.
 
-## Pulsar admin authentication
+### Pulsar admin authentication
 
 ```java
 String authPluginClassName = "com.org.MyAuthPluginClass";
diff --git a/site2/docs/security-overview.md b/site2/docs/security-overview.md
index 97c1ffb3aa..abc25261b4 100644
--- a/site2/docs/security-overview.md
+++ b/site2/docs/security-overview.md
@@ -27,12 +27,12 @@ role token is then used for [Authorization and ACLs](security-authorization.md)
 
 Currently Pulsar supports two authentication providers:
 
-- [TLS Authentication](security-tls.md)
+- [TLS Authentication](security-tls-authentication.md)
 - [Athenz](security-athenz.md)
 
 ## Contents
 
-- [Encryption and Authentication using TLS](security-tls.md)
+- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication.md)
using TLS
 - [Authentication using Athenz](security-athenz.md)
 - [Authorization and ACLs](security-authorization.md)
 - [End-to-End Encryption](security-encryption.md)
diff --git a/site2/docs/security-tls-authentication.md b/site2/docs/security-tls-authentication.md
new file mode 100644
index 0000000000..d5eb7d7ebe
--- /dev/null
+++ b/site2/docs/security-tls-authentication.md
@@ -0,0 +1,136 @@
+---
+id: security-tls-authentication
+title: Authentication using TLS
+sidebar_label: Authentication using TLS
+---
+
+## TLS Authentication Overview
+
+TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md),
but instead of only servers having keys and certs which the client uses the verify the server's
identity, clients also have keys and certs which the server uses to verify the client's identity.
You must have TLS transport encryption configured on your cluster before you can use TLS authentication.
This guide assumes you already have TLS transport encryption configured.
+
+### Creating client certificates
+
+Client certificates are generated using the same certificate authority as was used to generate
the server certificates.
+
+The biggest difference between client certs and server certs is that the **common name**
for the client certificate is the **role token** which that client will be authenticated as.
+
+First generate the key.
+```bash
+$ openssl genrsa -out admin.key.pem 2048
+```
+
+Similar to the broker, the client expects the key to be in [PKCS 8](https://en.wikipedia.org/wiki/PKCS_8)
format, so convert it.
+
+```bash
+$ openssl pkcs8 -topk8 -inform PEM -outform PEM \
+      -in admin.key.pem -out admin.key-pk8.pem -nocrypt
+```
+
+Generate the certificate request. When asked for a **common name**, enter the **role token**
which you want this key pair to authenticate a client as.
+
+```bash
+$ openssl req -config openssl.cnf \
+      -key admin.key.pem -new -sha256 -out admin.cert.pem
+```
+
+Sign with request with the certificate authority. Note that that client certs uses the **usr_cert**
extension, which allows the cert to be used for client authentication.
+
+```bash
+$ openssl ca -config openssl.cnf -extensions usr_cert \
+      -days 1000 -notext -md sha256 \
+      -in admin.csr.pem -out admin.cert.pem
+```
+
+This will give you a cert, `admin.cert.pem`, and a key, `admin.key-pk8.pem`, which, with
`ca.cert.pem`, can be used by clients to authenticate themselves to brokers and proxies as
the role token ``admin``.
+
+## Enabling TLS Authentication ...
+
+### ... on Brokers
+
+To configure brokers to authenticate clients, put the following in `broker.conf`, alongside
[the configuration to enable tls transport](security-tls-transport.md#broker-configuration):
+
+```properties
+# Configuration to enable authentication
+authenticationEnabled=true
+authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
+```
+
+### ... on Proxies
+
+To configure proxies to authenticate clients, put the folling in `proxy.conf`, alongside
[the configuration to enable tls transport](security-tls-transport.md#proxy-configuration):
+
+The proxy should have its own client key pair for connecting to brokers. The role token for
this key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization
guide](security-authorization.md) for more details.
+
+```properties
+# For clients connecting to the proxy
+authenticationEnabled=true
+authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
+
+# For the proxy to connect to brokers
+brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
+brokerClientAuthenticationParameters=tlsCertFile:/path/to/proxy.cert.pem,tlsKeyFile:/path/to/proxy.key-pk8.pem
+```
+
+## Client configuration
+
+When TLS authentication, the client needs to connect via TLS transport, so you need to configure
the client to use ```https://``` and port 8443 for the web service URL, and ```pulsar+ssl://```
and port 6651 for the broker service URL.
+
+### CLI tools
+
+[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md),
[`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client)
use the `conf/client.conf` config file in a Pulsar installation.
+
+You'll need to add the following parameters to that file to use TLS authentication with Pulsar's
CLI tools:
+
+```properties
+webServiceUrl=https://broker.example.com:8443/
+brokerServiceUrl=pulsar+ssl://broker.example.com:6651/
+useTls=true
+tlsAllowInsecureConnection=false
+tlsTrustCertsFilePath=/path/to/ca.cert.pem
+authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
+authParams=tlsCertFile:/path/to/my-role.cert.pem,tlsKeyFile:/path/to/my-role.key-pk8.pem
+```
+
+### Java client
+
+```java
+import org.apache.pulsar.client.api.PulsarClient;
+
+PulsarClient client = PulsarClient.builder()
+    .serviceUrl("pulsar+ssl://broker.example.com:6651/")
+    .enableTls(true)
+    .tlsTrustCertsFilePath("/path/to/ca.cert.pem")
+    .authentication("org.apache.pulsar.client.impl.auth.AuthenticationTls",
+                    "tlsCertFile:/path/to/my-role.cert.pem,tlsKeyFile:/path/to/my-role.key-pk8.pem")
+    .build();
+```
+
+### Python client
+
+```python
+from pulsar import Client, AuthenticationTLS
+
+auth = AuthenticationTLS("/path/to/my-role.cert.pem", "/path/to/my-role.key-pk8.pem")
+client = Client("pulsar+ssl://broker.example.com:6651/",
+                tls_trust_certs_file_path="/path/to/ca.cert.pem",
+                tls_allow_insecure_connection=False,
+				authentication=auth)
+```
+
+### C++ client
+
+```c++
+#include <pulsar/Client.h>
+
+pulsar::ClientConfiguration config;
+config.setUseTls(true);
+config.setTlsTrustCertsFilePath("/path/to/ca.cert.pem");
+config.setTlsAllowInsecureConnection(false);
+
+pulsar::AuthenticationPtr auth = pulsar::AuthTls::create("/path/to/my-role.cert.pem",
+                                                         "/path/to/my-role.key-pk8.pem")
+config.setAuth(auth);
+
+pulsar::Client client("pulsar+ssl://broker.example.com:6651/", config);
+```
+
diff --git a/site2/docs/security-tls-transport.md b/site2/docs/security-tls-transport.md
new file mode 100644
index 0000000000..70db61f0be
--- /dev/null
+++ b/site2/docs/security-tls-transport.md
@@ -0,0 +1,196 @@
+---
+id: security-tls-transport
+title: Transport Encryption using TLS
+sidebar_label: Transport Encryption using TLS
+---
+
+## TLS Overview
+
+By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text,
which means that all data is sent in the clear. TLS can be used to encrypt this traffic so
that it cannot be snooped by a man-in-the-middle attacker.
+
+TLS can be configured for both encryption and authentication. You may configure just TLS
transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md).
Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of
TLS transport encryption.
+
+> Note that enabling TLS may have a performance impact due to encryption overhead.
+
+## TLS concepts
+
+TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public-key_cryptography).
Encryption is performed using key pairs consisting of a public key and a private key. Messages
are encrypted with the public key and can be decrypted with the private key.
+
+To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and
a **certificate authority**.
+
+A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication.md).
+
+The **certificate authority** private key should be stored in a very secure location (a fully
encrypted, disconnected, air gapped computer). The certificate authority public key, the **trust
cert**, can be freely shared.
+
+For both client and server key pairs, the administrator first generates a private key and
a certificate request. Then the certificate authority private key is used to sign the certificate
request, generating a certificate. This certificate is the public key for the server/client
key pair.
+
+For TLS transport encryption, the clients can use the **trust cert** to verify that the server
they are talking to has a key pair that was signed by the certificate authority. A man-in-the-middle
attacker would not have access to the certificate authority, so they couldn't create a server
with such a key pair.
+
+For TLS authentication, the server uses the **trust cert** to verify that the client has
a key pair that was signed by the certificate authority. The Common Name of the **client cert**
is then used as the client's role token (see [Overview](security-overview.md)).
+
+## Creating TLS Certificates
+
+Creating TLS certificates for Pulsar involves creating a [certificate authority](#certificate-authority)
(CA), [server certificate](#server-certificate), and [client certificate](#client-certificate).
+
+The following guide is an abridged guide to setting up a certificate authority. For a more
detailed guide, there are plenty of resource on the internet. We recommend the [this guide](https://jamielinux.com/docs/openssl-certificate-authority/index.html).
+
+### Certificate authority
+
+The first step is to create the certificate for the CA. The CA will be used to sign both
the broker and client certificates, in order to ensure that each party will trust the others.
The CA should be stored in a very secure location (ideally completely disconnected from networks,
air gapped, and fully encrypted).
+
+Create a directory for your CA, and place [this openssl configuration file](https://github.com/apache/incubator-pulsar/tree/master/site2/website/static/examples/openssl.cnf)
in the directory. You may want to modify the default answers for company name and department
in the configuration file. Export the location of the CA directory to the environment variable,
CA_HOME. The configuration file uses this environment variable to find the rest of the files
and directories needed for the CA.
+
+```bash
+$ mkdir my-ca
+$ cd my-ca
+$ wget https://github.com/apache/incubator-pulsar/tree/master/site2/website/static/examples/openssl.cnf
+$ export CA_HOME=$(pwd)
+```
+
+Create the necessary directories, keys and certs.
+
+```bash
+$ mkdir certs crl newcerts private
+$ chmod 700 private/
+$ touch index.txt
+$ echo 1000 > serial
+$ openssl genrsa -aes256 -out private/ca.key.pem 4096
+$ chmod 400 private/ca.key.pem
+$ openssl req -config openssl.cnf -key private/ca.key.pem \
+      -new -x509 -days 7300 -sha256 -extensions v3_ca \
+      -out certs/ca.cert.pem
+$ chmod 444 certs/ca.cert.pem
+```
+
+After answering the question prompts, this will store CA-related files in the `./my-ca` directory.
Within that directory:
+
+* `certs/ca.cert.pem` is the public certificate. It is meant to be distributed to all parties
involved.
+* `private/ca.key.pem` is the private key. This is only needed when signing a new certificate
for either broker or clients and it must be safely guarded.
+
+### Server certificate
+
+Once a CA certificate has been created, you can create certificate requests and sign them
with the CA.
+
+The following commands will ask you a few questions and then create the certificates. When
asked for the common name, you should match the hostname of the broker. You could also use
a wildcard to match a group of broker hostnames, for example `*.broker.usw.example.com`. This
ensures that the same certificate can be reused on multiple machines.
+
+> #### Tips
+> 
+> Sometimes it is not possible or makes no sense to match the hostname,
+> such as when the brokers are created with random hostnames, or you
+> plan to connect to the hosts via their IP. In this case, the client
+> should be configured to disable TLS hostname verification.
+
+First generate the key.
+```bash
+$ openssl genrsa -out broker.key.pem 2048
+```
+
+The broker expects the key to be in [PKCS 8](https://en.wikipedia.org/wiki/PKCS_8) format,
so convert it.
+
+```bash
+$ openssl pkcs8 -topk8 -inform PEM -outform PEM \
+      -in broker.key.pem -out broker.key-pk8.pem -nocrypt
+```
+
+Generate the certificate request...
+
+```bash
+$ openssl req -config openssl.cnf \
+      -key broker.key.pem -new -sha256 -out broker.cert.pem
+```
+
+... and sign it with the certificate authority.
+```bash
+$ openssl ca -config openssl.cnf -extensions server_cert \
+      -days 1000 -notext -md sha256 \
+      -in broker.csr.pem -out broker.cert.pem
+```
+
+At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pem`, which
can be used along with `ca.cert.pem` to configure TLS transport encryption for your broker
and proxy nodes.
+
+## Broker Configuration
+
+To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption,
you'll need to make some changes to `broker.conf`, which is located in the `conf` directory
of your [Pulsar installation](getting-started-standalone.md).
+
+Add these values to the configuration file (substituting the appropriate certificate paths
where necessary):
+
+```properties
+tlsEnabled=true
+tlsCertificateFilePath=/path/to/broker.cert.pem
+tlsKeyFilePath=/path/to/broker.key-pk8.pem
+tlsTrustCertsFilePath=/path/to/ca.cert.pem
+```
+
+> A full list of parameters available in the `conf/broker.conf` file,
+> as well as the default values for those parameters, can be found in [Broker Configuration](reference-configuration.md#broker)

+
+## Proxy Configuration
+
+Proxies need to configure TLS in two directions, for clients connecting to the proxy, and
for the proxy to be able to connect to brokers.
+
+```properties
+# For clients connecting to the proxy
+tlsEnabledInProxy=true
+tlsCertificateFilePath=/path/to/broker.cert.pem
+tlsKeyFilePath=/path/to/broker.key-pk8.pem
+tlsTrustCertsFilePath=/path/to/ca.cert.pem
+
+# For the proxy to connect to brokers
+tlsEnabledWithBroker=true
+brokerClientTrustCertsFilePath=/path/to/ca.cert.pem
+```
+
+## Client configuration
+
+When TLS transport encryption is enabled, you need to configure the client to use ```https://```
and port 8443 for the web service URL, and ```pulsar+ssl://``` and port 6651 for the broker
service URL.
+
+As the server certificate you generated above doesn't belong to any of the default trust
chains, you also need to either specify the path the **trust cert** (recommended), or tell
the client to allow untrusted server certs.
+
+### CLI tools
+
+[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin),
[`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client)
use the `conf/client.conf` config file in a Pulsar installation.
+
+You'll need to add the following parameters to that file to use TLS transport with Pulsar's
CLI tools:
+
+```properties
+webServiceUrl=https://broker.example.com:8443/
+brokerServiceUrl=pulsar+ssl://broker.example.com:6651/
+useTls=true
+tlsAllowInsecureConnection=false
+tlsTrustCertsFilePath=/path/to/ca.cert.pem
+```
+
+### Java client
+
+```java
+import org.apache.pulsar.client.api.PulsarClient;
+
+PulsarClient client = PulsarClient.builder()
+    .serviceUrl("pulsar+ssl://broker.example.com:6651/")
+    .enableTls(true)
+    .tlsTrustCertsFilePath("/path/to/ca.cert.pem")
+    .build();
+```
+
+### Python client
+
+```python
+from pulsar import Client
+
+client = Client("pulsar+ssl://broker.example.com:6651/",
+                tls_trust_certs_file_path="/path/to/ca.cert.pem",
+                tls_allow_insecure_connection=False)
+```
+
+### C++ client
+
+```c++
+#include <pulsar/Client.h>
+
+pulsar::ClientConfiguration config;
+config.setUseTls(true);
+config.setTlsTrustCertsFilePath("/path/to/ca.cert.pem");
+config.setTlsAllowInsecureConnection(false);
+
+pulsar::Client client("pulsar+ssl://broker.example.com:6651/", config);
+```
diff --git a/site2/docs/security-tls.md b/site2/docs/security-tls.md
deleted file mode 100644
index 190b5c8f06..0000000000
--- a/site2/docs/security-tls.md
+++ /dev/null
@@ -1,142 +0,0 @@
----
-id: security-tls
-title: Encryption and Authentication using TLS
-sidebar_label: Encryption and Authentication using TLS
----
-
-With [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) authentication, the server
authenticates the client (also called “2-way authentication”).
-Since TLS authentication requires TLS encryption, this page shows you how to configure both
at the same time.
-
-By default, Apache Pulsar communicates in plain text service url, which means that all data
is sent in the clear.
-To encrypt communication, it is recommended to configure all the Apache Pulsar components
in your deployment to use TLS encryption.
-
-TLS can be configured for encryption or authentication. You may configure just TLS encryption
-(by default TLS encryption includes certificate authentication of the server) and independently
choose a separate mechanism
-for client authentication, e.g. TLS, [Athenz](security-athenz.md), etc. Note that TLS encryption,
technically speaking, already enables
-1-way authentication in which the client authenticates the server certificate. So when referring
to TLS authentication, it is really
-referring to 2-way authentication in which the broker also authenticates the client certificate.
-
-> Note that enabling TLS may have a performance impact due to encryption overhead.
-
-## Creating TLS Certificates
-
-Creating TLS certificates for Pulsar involves creating a [certificate authority](#certificate-authority)
(CA), [broker certificate](#broker-certificate), and [client certificate](#client-certificate).
-
-### Certificate authority
-
-The first step is to create the certificate for the CA. The CA will be used to sign both
the broker and client certificates, in order to ensure that each party will trust the others.
-
-#### Linux
-
-```bash
-$ CA.pl -newca
-```
-
-#### macOS
-
-```bash
-$ /System/Library/OpenSSL/misc/CA.pl -newca
-```
-
-After answering the question prompts, this will store CA-related files in the `./demoCA`
directory. Within that directory:
-
-* `demoCA/cacert.pem` is the public certificate. It is meant to be distributed to all parties
involved.
-* `demoCA/private/cakey.pem` is the private key. This is only needed when signing a new certificate
for either broker or clients and it must be safely guarded.
-
-### Broker certificate
-
-Once a CA certificate has been created, you can create certificate requests and sign them
with the CA.
-
-The following commands will ask you a few questions and then create the certificates. When
asked for the common name, you need to match the hostname of the broker. You could also use
a wildcard to match a group of broker hostnames, for example `*.broker.usw.example.com`. This
ensures that the same certificate can be reused on multiple machines.
-
-```shell
-$ openssl req \
-  -newkey rsa:2048 \
-  -sha256 \
-  -nodes \
-  -out broker-cert.csr \
-  -outform PEM
-```
-
-Convert the key to [PKCS 8](https://en.wikipedia.org/wiki/PKCS_8) format:
-
-```shell
-$ openssl pkcs8 \
-  -topk8 \
-  -inform PEM \
-  -outform PEM \
-  -in privkey.pem \
-  -out broker-key.pem \
-  -nocrypt
-```
-
-This will create two broker certificate files named `broker-cert.csr` and `broker-key.pem`.
Now you can create the signed certificate:
-
-```shell
-$ openssl ca \
-  -out broker-cert.pem \
-  -infiles broker-cert.csr
-```
-
-At this point, you should have a `broker-cert.pem` and `broker-key.pem` file. These will
be needed for the broker.
-
-### Client certificate
-
-To create a client certificate, repeat the steps in the previous section, but did create
`client-cert.pem` and `client-key.pem` files instead.
-
-For the client common name, you need to use a string that you intend to use as the [role
token](security-overview.md#role-tokens) for this client, though it doesn't need to match
the client hostname.
-
-## Configure the broker for TLS
-
-To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS authentication,
you'll need to make some changes to the `broker.conf` configuration file, which is located
in the `conf` directory of your [Pulsar installation](getting-started-standalone.md).
-
-Add these values to the configuration file (substituting the appropriate certificate paths
where necessary):
-
-```properties
-# Enable TLS and point the broker to the right certs
-tlsEnabled=true
-tlsCertificateFilePath=/path/to/broker-cert.pem
-tlsKeyFilePath=/path/to/broker-key.pem
-tlsTrustCertsFilePath=/path/to/cacert.pem
-
-# Enable the TLS auth provider
-authenticationEnabled=true
-authorizationEnabled=true
-authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
-```
-
-> A full listing of parameters available in the `conf/broker.conf` file, as well as the
default
-> values for those parameters, can be found in [Broker Configuration](reference-configuration.md#broker).
-
-## Configure the discovery service
-
-The [discovery service](reference-terminology.md#service-discovery) used by Pulsar brokers
needs to redirect all HTTPS requests, which means that it needs to be trusted by the client
as well. Add this configuration in `conf/discovery.conf` in your Pulsar installation:
-
-```properties
-tlsEnabled=true
-tlsCertificateFilePath=/path/to/broker-cert.pem
-tlsKeyFilePath=/path/to/broker-key.pem
-```
-
-## Configure clients
-
-For more information on Pulsar client authentication using TLS, see the following language-specific
docs:
-
-* [Java client](client-libraries-java.md)
-* [C++ client](client-libraries-cpp.md)
-
-## Configure CLI tools
-
-[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md),
[`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client)
use the `conf/client.conf` config file in a Pulsar installation.
-
-You'll need to add the following authentication parameters to that file to use TLS with Pulsar's
CLI tools:
-
-```properties
-serviceUrl=https://broker.example.com:8443/
-authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
-authParams=tlsCertFile:/path/to/client-cert.pem,tlsKeyFile:/path/to/client-key.pem
-useTls=true
-tlsAllowInsecureConnection=false
-tlsTrustCertsFilePath=/path/to/cacert.pem
-```
-
diff --git a/site2/website/sidebars.json b/site2/website/sidebars.json
index 48a50d43da..c852fee2fa 100644
--- a/site2/website/sidebars.json
+++ b/site2/website/sidebars.json
@@ -48,7 +48,8 @@
     ],
     "Security": [
       "security-overview",
-      "security-tls",
+      "security-tls-transport",
+      "security-tls-authentication",
       "security-athenz",
       "security-authorization",
       "security-encryption",
diff --git a/site2/website/static/examples/openssl.cnf b/site2/website/static/examples/openssl.cnf
new file mode 100644
index 0000000000..eee8a47298
--- /dev/null
+++ b/site2/website/static/examples/openssl.cnf
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# 
+# Configuration file for testing certificate authority.
+# The environment variable, CA_HOME, must be set to point to the directory
+# containing this file before running any openssl commands.
+#
+[ ca ]
+# `man ca`
+default_ca = CA_default
+
+[ CA_default ]
+# Directory and file locations.
+dir               = $ENV::CA_HOME
+certs             = $dir/certs
+crl_dir           = $dir/crl
+new_certs_dir     = $dir/newcerts
+database          = $dir/index.txt
+serial            = $dir/serial
+RANDFILE          = $dir/private/.rand
+
+# The root key and root certificate.
+private_key       = $dir/private/ca.key.pem
+certificate       = $dir/certs/ca.cert.pem
+
+# For certificate revocation lists.
+crlnumber         = $dir/crlnumber
+crl               = $dir/crl/ca.crl.pem
+crl_extensions    = crl_ext
+default_crl_days  = 30
+
+# SHA-1 is deprecated, so use SHA-2 instead.
+default_md        = sha256
+
+name_opt          = ca_default
+cert_opt          = ca_default
+default_days      = 375
+preserve          = no
+policy            = policy_strict
+
+[ policy_strict ]
+# The root CA should only sign intermediate certificates that match.
+# See the POLICY FORMAT section of `man ca`.
+countryName             = match
+stateOrProvinceName     = match
+organizationName        = match
+organizationalUnitName  = optional
+commonName              = supplied
+emailAddress            = optional
+
+[ policy_loose ]
+# Allow the intermediate CA to sign a more diverse range of certificates.
+# See the POLICY FORMAT section of the `ca` man page.
+countryName             = optional
+stateOrProvinceName     = optional
+localityName            = optional
+organizationName        = optional
+organizationalUnitName  = optional
+commonName              = supplied
+emailAddress            = optional
+
+[ req ]
+# Options for the `req` tool (`man req`).
+default_bits        = 2048
+distinguished_name  = req_distinguished_name
+string_mask         = utf8only
+
+# SHA-1 is deprecated, so use SHA-2 instead.
+default_md          = sha256
+
+# Extension to add when the -x509 option is used.
+x509_extensions     = v3_ca
+
+[ req_distinguished_name ]
+# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
+countryName                     = Country Name (2 letter code)
+stateOrProvinceName             = State or Province Name
+localityName                    = Locality Name
+0.organizationName              = Organization Name
+organizationalUnitName          = Organizational Unit Name
+commonName                      = Common Name
+emailAddress                    = Email Address
+
+# Optionally, specify some defaults.
+countryName_default             = US
+stateOrProvinceName_default     = California
+localityName_default            = Palo Alto
+0.organizationName_default      = My company
+organizationalUnitName_default  = IT
+emailAddress_default            =
+
+[ v3_ca ]
+# Extensions for a typical CA (`man x509v3_config`).
+subjectKeyIdentifier = hash
+authorityKeyIdentifier = keyid:always,issuer
+basicConstraints = critical, CA:true
+keyUsage = critical, digitalSignature, cRLSign, keyCertSign
+
+[ v3_intermediate_ca ]
+# Extensions for a typical intermediate CA (`man x509v3_config`).
+subjectKeyIdentifier = hash
+authorityKeyIdentifier = keyid:always,issuer
+basicConstraints = critical, CA:true, pathlen:0
+keyUsage = critical, digitalSignature, cRLSign, keyCertSign
+
+[ usr_cert ]
+# Extensions for client certificates (`man x509v3_config`).
+basicConstraints = CA:FALSE
+nsCertType = client, email
+nsComment = "OpenSSL Generated Client Certificate"
+subjectKeyIdentifier = hash
+authorityKeyIdentifier = keyid,issuer
+keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
+extendedKeyUsage = clientAuth, emailProtection
+
+[ server_cert ]
+# Extensions for server certificates (`man x509v3_config`).
+basicConstraints = CA:FALSE
+nsCertType = server
+nsComment = "OpenSSL Generated Server Certificate"
+subjectKeyIdentifier = hash
+authorityKeyIdentifier = keyid,issuer:always
+keyUsage = critical, digitalSignature, keyEncipherment
+extendedKeyUsage = serverAuth
+
+[ crl_ext ]
+# Extension for CRLs (`man x509v3_config`).
+authorityKeyIdentifier=keyid:always
+
+[ ocsp ]
+# Extension for OCSP signing certificates (`man ocsp`).
+basicConstraints = CA:FALSE
+subjectKeyIdentifier = hash
+authorityKeyIdentifier = keyid,issuer
+keyUsage = critical, digitalSignature
+extendedKeyUsage = critical, OCSPSigning
\ No newline at end of file


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message