- Encryption and Authentication using TLS + Transport Encryption using TLS
@@ -987,103 +994,137 @@TLS Overview
-With TLS authentication, the server authenticates the client (also called “2-way authentication”). -Since TLS authentication requires TLS encryption, this page shows you how to configure both at the same time.
- -By default, Apache Pulsar communicates in plain text service url, which means that all data is sent in the clear. -To encrypt communication, it is recommended to configure all the Apache Pulsar components in your deployment to use TLS encryption.
+By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text, which means that all data is sent in the clear. TLS can be used to encrypt this traffic so that it cannot be snooped by a man-in-the-middle attacker.
-TLS can be configured for encryption or authentication. You may configure just TLS encryption -(by default TLS encryption includes certificate authentication of the server) and independently choose a separate mechanism -for client authentication, e.g. TLS, Athenz, etc. Note that TLS encryption, technically speaking, already enables -1-way authentication in which the client authenticates the server certificate. So when referring to TLS authentication, it is really -referring to 2-way authentication in which the broker also authenticates the client certificate.
+TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered elsewhere. Alternatively, you can use another authentication mechanism on top of TLS transport encryption.
+Note that enabling TLS may have a performance impact due to encryption overhead.
TLS concepts
+ +TLS is a form of public key cryptography. Encryption is performed using key pairs consisting of a public key and a private key. Messages are encrypted with the public key and can be decrypted with the private key.
+ +To use TLS transport encryption, you need two kinds of key pairs, server key pairs and a certificate authority.
+ +A third kind of key pair, client key pairs, are used for client authentication.
+ +The certificate authority private key should be stored in a very secure location (a fully encrypted, disconnected, air gapped computer). The certificate authority public key, the trust cert, can be freely shared.
+ +For both client and server key pairs, the administrator first generates a private key and a certificate request. Then the certificate authority private key is used to sign the certificate request, generating a certificate. This certificate is the public key for the server/client key pair.
+ +For TLS transport encryption, the clients can use the trust cert to verify that the server they are talking to has a key pair that was signed by the certificate authority. A man-in-the-middle attacker would not have access to the certificate authority, so they couldn’t create a server with such a key pair.
+ +For TLS authentication, the server uses the trust cert to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the client cert is then used as the client’s role token (see Overview).
+Creating TLS Certificates
-Creating TLS certificates for Pulsar involves creating a certificate authority (CA), broker certificate, and client certificate.
+Creating TLS certificates for Pulsar involves creating a certificate authority (CA), server certificate, and client certificate.
+ +The following guide is an abridged guide to setting up a certificate authority. For a more detailed guide, there are plenty of resource on the internet. We recommend the this guide.
Certificate authority
-The first step is to create the certificate for the CA. The CA will be used to sign both the broker and client certificates, in order to ensure that each party will trust the others.
+The first step is to create the certificate for the CA. The CA will be used to sign both the broker and client certificates, in order to ensure that each party will trust the others. The CA should be stored in a very secure location (ideally completely disconnected from networks, air gapped, and fully encrypted).
-Linux
+Create a directory for your CA, and place this openssl configuration file in the directory. You may want to modify the default answers for company name and department in the configuration file. Export the location of the CA directory to the environment variable, CA_HOME. The configuration file uses this environment variable to find the rest of the files and directories needed for the CA.
-$ CA.pl -newca
+$ mkdir my-ca
+$ cd my-ca
+$ wget /docs/latest/security/openssl.cnf
+$ export CA_HOME=$(pwd)
-macOS
-
-$ /System/Library/OpenSSL/misc/CA.pl -newca
+Create the necessary directories, keys and certs.
+
+$ mkdir certs crl newcerts private
+$ chmod 700 private/
+$ touch index.txt
+$ echo 1000 > serial
+$ openssl genrsa -aes256 -out private/ca.key.pem 4096
+$ chmod 400 private/ca.key.pem
+$ openssl req -config openssl.cnf -key private/ca.key.pem \
+ -new -x509 -days 7300 -sha256 -extensions v3_ca \
+ -out certs/ca.cert.pem
+$ chmod 444 certs/ca.cert.pem
-After answering the question prompts, this will store CA-related files in the ./demoCA
directory. Within that directory:
+After answering the question prompts, this will store CA-related files in the ./my-ca
directory. Within that directory:
- demoCA/cacert.pem
is the public certificate. It is meant to be distributed to all parties involved.
- demoCA/private/cakey.pem
is the private key. This is only needed when signing a new certificate for either broker or clients and it must be safely guarded.
+ certs/ca.cert.pem
is the public certificate. It is meant to be distributed to all parties involved.
+ private/ca.key.pem
is the private key. This is only needed when signing a new certificate for either broker or clients and it must be safely guarded.
-Broker certificate
+Server certificate
Once a CA certificate has been created, you can create certificate requests and sign them with the CA.
-The following commands will ask you a few questions and then create the certificates. When asked for the common name, you need to match the hostname of the broker. You could also use a wildcard to match a group of broker hostnames, for example *.broker.usw.example.com
. This ensures that the same certificate can be reused on multiple machines.
+The following commands will ask you a few questions and then create the certificates. When asked for the common name, you should match the hostname of the broker. You could also use a wildcard to match a group of broker hostnames, for example *.broker.usw.example.com
. This ensures that the same certificate can be reused on multiple machines.
-$ openssl req \
- -newkey rsa:2048 \
- -sha256 \
- -nodes \
- -out broker-cert.csr \
- -outform PEM
-
+
+
+
+
+ Sometimes it is not possible or makes no sense to match the hostname, such as when the brokers are created with random hostnames, or you plan to connect to the hosts via their IP. In this case, the client should be configured to disable TLS hostname verification.
-Convert the key to PKCS 8 format:
+
+
-$ openssl pkcs8 \
- -topk8 \
- -inform PEM \
- -outform PEM \
- -in privkey.pem \
- -out broker-key.pem \
- -nocrypt
+First generate the key.
+$ openssl genrsa -out broker.key.pem 2048
-This will create two broker certificate files named broker-cert.csr
and broker-key.pem
. Now you can create the signed certificate:
+The broker expects the key to be in PKCS 8 format, so convert it.
-$ openssl ca \
- -out broker-cert.pem \
- -infiles broker-cert.csr
+$ openssl pkcs8 -topk8 -inform PEM -outform PEM \
+ -in broker.key.pem -out broker.key-pk8.pem -nocrypt
-At this point, you should have a broker-cert.pem
and broker-key.pem
file. These will be needed for the broker.
+Generate the certificate request…
-Client certificate
+$ openssl req -config openssl.cnf \
+ -key broker.key.pem -new -sha256 -out broker.cert.pem
+
-To create a client certificate, repeat the steps in the previous section, but did create client-cert.pem
and client-key.pem
files instead.
+… and sign it with the certificate authority.
+$ openssl ca -config openssl.cnf -extensions server_cert \
+ -days 1000 -notext -md sha256 \
+ -in broker.csr.pem -out broker.cert.pem
+
-For the client common name, you need to use a string that you intend to use as the role token for this client, though it doesn’t need to match the client hostname.
+At this point, you have a cert, broker.cert.pem
, and a key, broker.key-pk8.pem
, which can be used along with ca.cert.pem
to configure TLS transport encryption for your broker and proxy nodes.
-Configure the broker for TLS
+Broker Configuration
-To configure a Pulsar broker to use TLS authentication, you’ll need to make some [...]
+
To configure a Pulsar broker to use TLS transport encryption, you’ll need to mak [...]
Add these values to the configuration file (substituting the appropriate certificate paths where necessary):
-# Enable TLS and point the broker to the right certs
-tlsEnabled=true
-tlsCertificateFilePath=/path/to/broker-cert.pem
-tlsKeyFilePath=/path/to/broker-key.pem
-tlsTrustCertsFilePath=/path/to/cacert.pem
-
-# Enable the TLS auth provider
-authenticationEnabled=true
-authorizationEnabled=true
-authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
+tlsEnabled=true
+tlsCertificateFilePath=/path/to/broker.cert.pem
+tlsKeyFilePath=/path/to/broker.key-pk8.pem
+tlsTrustCertsFilePath=/path/to/ca.cert.pem
-Encryption and Authentication using TLS
+Authentication using TLS
@@ -565,8 +565,15 @@
+
+ Transport Encryption with TLS
+
+
+
+
+
- Encryption and Authentication using TLS
+ Authentication using TLS
@@ -951,7 +958,7 @@
- Encryption and Authentication using TLS
+ Authentication using TLS
@@ -985,175 +992,134 @@
-->
-TLS Overview
-
-With TLS authentication, the server authenticates the client (also called “2-way authentication”).
-Since TLS authentication requires TLS encryption, this page shows you how to configure both at the same time.
-
-By default, Apache Pulsar communicates in plain text service url, which means that all data is sent in the clear.
-To encrypt communication, it is recommended to configure all the Apache Pulsar components in your deployment to use TLS encryption.
+TLS Authentication Overview
-TLS can be configured for encryption or authentication. You may configure just TLS encryption
-(by default TLS encryption includes certificate authentication of the server) and independently choose a separate mechanism
-for client authentication, e.g. TLS, Athenz, etc. Note that TLS encryption, technically speaking, already enables
-1-way authentication in which the client authenticates the server certificate. So when referring to TLS authentication, it is really
-referring to 2-way authentication in which the broker also authenticates the client certificate.
+TLS authentication is an extension of TLS transport encryption, but instead of only servers having keys and certs which the client uses the verify the server’s identity, clients also have keys and certs which the server uses to verify the client’s identity. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured.
-
- Note that enabling TLS may have a performance impact due to encryption overhead.
-
+Creating client certificates
-Creating TLS Certificates
-
-Creating TLS certificates for Pulsar involves creating a certificate authority (CA), broker certificate, and client certificate.
-
-Certificate authority
-
-The first step is to create the certificate for the CA. The CA will be used to sign both the broker and client certificates, in order to ensure that each party will trust the others.
-
-Linux
-
-$ CA.pl -newca
-
+Client certificates are generated using the same certificate authority as was used to generate the server certificates.
-macOS
+The biggest difference between client certs and server certs is that the common name for the client certificate is the role token which that client will be authenticated as.
-$ /System/Library/OpenSSL/misc/CA.pl -newca
+First generate the key.
+$ openssl genrsa -out admin.key.pem 2048
-After answering the question prompts, this will store CA-related files in the ./demoCA
directory. Within that directory:
+Similar to the broker, the client expects the key to be in PKCS 8 format, so convert it.
-
- demoCA/cacert.pem
is the public certificate. It is meant to be distributed to all parties involved.
- demoCA/private/cakey.pem
is the private key. This is only needed when signing a new certificate for either broker or clients and it must be safely guarded.
-
-
-Broker certificate
-
-Once a CA certificate has been created, you can create certificate requests and sign them with the CA.
-
-The following commands will ask you a few questions and then create the certificates. When asked for the common name, you need to match the hostname of the broker. You could also use a wildcard to match a group of broker hostnames, for example *.broker.usw.example.com
. This ensures that the same certificate can be reused on multiple machines.
-
-$ openssl req \
- -newkey rsa:2048 \
- -sha256 \
- -nodes \
- -out broker-cert.csr \
- -outform PEM
+$ openssl pkcs8 -topk8 -inform PEM -outform PEM \
+ -in admin.key.pem -out admin.key-pk8.pem -nocrypt
-Convert the key to PKCS 8 format:
+Generate the certificate request. When asked for a common name, enter the role token which you want this key pair to authenticate a client as.
-$ openssl pkcs8 \
- -topk8 \
- -inform PEM \
- -outform PEM \
- -in privkey.pem \
- -out broker-key.pem \
- -nocrypt
+$ openssl req -config openssl.cnf \
+ -key admin.key.pem -new -sha256 -out admin.cert.pem
-This will create two broker certificate files named broker-cert.csr
and broker-key.pem
. Now you can create the signed certificate:
+Sign with request with the certificate authority. Note that that client certs uses the usr_cert extension, which allows the cert to be used for client authentication.
-$ openssl ca \
- -out broker-cert.pem \
- -infiles broker-cert.csr
+$ openssl ca -config openssl.cnf -extensions usr_cert \
+ -days 1000 -notext -md sha256 \
+ -in admin.csr.pem -out admin.cert.pem
-At this point, you should have a broker-cert.pem
and broker-key.pem
file. These will be needed for the broker.
+This will give you a cert, admin.cert.pem
, and a key, admin.key-pk8.pem
, which, with ca.cert.pem
, can be used by clients to authenticate themselves to brokers and proxies as the role token admin
.
-Client certificate
+Enabling TLS Authentication …
-To create a client certificate, repeat the steps in the previous section, but did create client-cert.pem
and client-key.pem
files instead.
+… on Brokers
-For the client common name, you need to use a string that you intend to use as the role token for this client, though it doesn’t need to match the client hostname.
+To configure brokers to authenticate clients, put the following in broker.conf
, alongside the configuration to enable tls transport:
-Configure the broker for TLS
+# Configuration to enable authentication
+authenticationEnabled=true
+authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
+
-To configure a Pulsar broker to use TLS authentication, you’ll need to make some [...]
+
… on Proxies
-Add these values to the configuration file (substituting the appropriate certificate paths where necessary):
+To configure proxies to authenticate clients, put the folling in proxy.conf
, alongside the configuration to enable tls transport:
-# Enable TLS and point the broker to the right certs
-tlsEnabled=true
-tlsCertificateFilePath=/path/to/broker-cert.pem
-tlsKeyFilePath=/path/to/broker-key.pem
-tlsTrustCertsFilePath=/path/to/cacert.pem
+The proxy should have its own client key pair for connecting to brokers. The role token for this key pair should be configured in the proxyRoles
of the brokers. See the authorization guide for more details.
-# Enable the TLS auth provider
+# For clients connecting to the proxy
authenticationEnabled=true
-authorizationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
-
-
+Command-line tools like pulsar-admin
, pulsar-perf
, and pulsar-client
use the conf/client.conf
config file in a Pulsar installation.
-
-
-
- A full listing of parameters available in the conf/broker.conf
file, as well as the default values for those parameters, can be found in Broker Configuration.
+You’ll need to add the following parameters to that file to use TLS authentication with Pulsar’s CLI tools:
-
-
+webServiceUrl=https://broker.example.com:8443/
+brokerServiceUrl=pulsar+ssl://broker.example.com:6651/
+useTls=true
+tlsAllowInsecureConnection=false
+tlsTrustCertsFilePath=/path/to/ca.cert.pem
+authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
+authParams=tlsCertFile:/path/to/my-role.cert.pem,tlsKeyFile:/path/to/my-role.key-pk8.pem
+
-Configure the discovery service
+Java client
-The discovery service used by Pulsar brokers needs to redirect all HTTPS requests, which means that it needs to be trusted by the client as well. Add this configuration in import org.apache.pulsar.client.api.PulsarClient;
-tlsEnabled=true
-tlsCertificateFilePath=/path/to/broker-cert.pem
-tlsKeyFilePath=/path/to/broker-key.pem
+PulsarClient client = PulsarClient.builder()
+ .serviceUrl("pulsar+ssl://broker.example.com:6651/")
+ .enableTls(true)
+ .tlsTrustCertsFilePath("/path/to/ca.cert.pem")
+ .authentication("org.apache.pulsar.client.impl.auth.AuthenticationTls",
+ "tlsCertFile:/path/to/my-role.cert.pem,tlsKeyFile:/path/to/my-role.key-pk8.pem")
+ .build();
-Configure clients
+Python client
-For more information on Pulsar client authentication using TLS, see the following language-specific docs:
+from pulsar import Client, AuthenticationTLS
-
- - Java client
- - C++ client
-
+auth = AuthenticationTLS("/path/to/my-role.cert.pem", "/path/to/my-role.key-pk8.pem")
+client = Client("pulsar+ssl://broker.example.com:6651/",
+ tls_trust_certs_file_path="/path/to/ca.cert.pem",
+ tls_allow_insecure_connection=False,
+ authentication=auth)
+
-Configure CLI tools
+C++ client
-Command-line tools like pulsar-admin
, pulsar-perf
, and pulsar-client
use the conf/client.conf
config file in a Pulsar installation.
+#include <pulsar/Client.h>
+
+pulsar::ClientConfiguration config;
+config.setUseTls(true);
+config.setTlsTrustCertsFilePath("/path/to/ca.cert.pem");
+config.setTlsAllowInsecureConnection(false);
-You’ll need to add the following authentication parameters to that file to use TLS with Pulsar’s CLI tools:
+pulsar::AuthenticationPtr auth = pulsar::AuthTls::create("/path/to/my-role.cert.pem",
+ "/path/to/my-role.key-pk8.pem")
+config.setAuth(auth);
-serviceUrl=https://broker.example.com:8443/
-authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
-authParams=tlsCertFile:/path/to/client-cert.pem,tlsKeyFile:/path/to/client-key.pem
-useTls=true
-tlsAllowInsecureConnection=false
-tlsTrustCertsFilePath=/path/to/cacert.pem
+pulsar::Client client("pulsar+ssl://broker.example.com:6651/", config);
+
diff --git a/content/ja/adaptors/PulsarSpark/index.html b/content/ja/adaptors/PulsarSpark/index.html
index 65c9ef7..34dd969 100644
--- a/content/ja/adaptors/PulsarSpark/index.html
+++ b/content/ja/adaptors/PulsarSpark/index.html
@@ -762,9 +762,11 @@
+ Spark Streaming Pulsar Receiver
+
+
- Spark Streaming Pulsar Receiver
@@ -1006,9 +1008,11 @@
+ Spark Streaming Pulsar Receiver
+
+
- Spark Streaming Pulsar Receiver
@@ -1220,8 +1224,6 @@
-
-
Pulsar Javaクライアント
@@ -1381,6 +1383,10 @@
+
+
+
+
diff --git a/content/ja/adaptors/PulsarStorm/index.html b/content/ja/adaptors/PulsarStorm/index.html
index 681e054..d24d155 100644
--- a/content/ja/adaptors/PulsarStorm/index.html
+++ b/content/ja/adaptors/PulsarStorm/index.html
@@ -766,9 +766,11 @@
+ Apache StormのためのPulsarアダプタ
+
+
- Apache StormのためのPulsarアダプタ
@@ -976,8 +978,6 @@
-
-
Pulsar Javaクライアント
@@ -1137,6 +1137,10 @@
+
+
+
+
diff --git a/content/ja/admin/AdminInterface/index.html b/content/ja/admin/AdminInterface/index.html
index 153dc3f..5a8fbe4 100644
--- a/content/ja/admin/AdminInterface/index.html
+++ b/content/ja/admin/AdminInterface/index.html
@@ -692,9 +692,9 @@
+ Pulsarコマンドラインツール
- Pulsarコマンドラインツール
@@ -762,9 +762,9 @@
+ パーティションドトピック
- パーティションドトピック
@@ -792,9 +792,11 @@
+ メッセージの保存と有効期限
+
+
- メッセージの保存と有効期限
@@ -948,9 +950,11 @@
+ Pulsarコマンドラインツール
+
+
- Pulsarコマンドラインツール
@@ -1389,6 +1393,8 @@
+
+
@@ -1480,8 +1486,6 @@
-
-
Pulsar Javaクライアント
@@ -1641,6 +1645,10 @@
+
+
+
+
diff --git a/content/ja/admin/Authz/index.html b/content/ja/admin/Authz/index.html
index c208d6e..910e23f 100644
--- a/content/ja/admin/Authz/index.html
+++ b/content/ja/admin/Authz/index.html
@@ -692,9 +692,9 @@
+ Pulsarコマンドラインツール
- Pulsarコマンドラインツール
@@ -762,9 +762,9 @@
+ パーティションドトピック
- パーティションドトピック
@@ -792,9 +792,11 @@
+ メッセージの保存と有効期限
+
+
- メッセージの保存と有効期限
@@ -1143,6 +1145,8 @@
+
+
@@ -1387,6 +1391,8 @@
+
+
@@ -1631,6 +1637,8 @@
+
+
@@ -1875,6 +1883,8 @@
+
+
@@ -1966,8 +1976,6 @@
-
-
Pulsar Javaクライアント
@@ -2127,6 +2135,10 @@
+
+
+
+
@@ -2200,9 +2212,11 @@
+ Pulsar C++クライアント
+
+
- Pulsar C++クライアント
diff --git a/content/ja/admin/ClustersBrokers/index.html b/content/ja/admin/ClustersBrokers/index.html
index 5b03a7e..ad4bdb4 100644
--- a/content/ja/admin/ClustersBrokers/index.html
+++ b/content/ja/admin/ClustersBrokers/index.html
@@ -692,9 +692,9 @@
+ Pulsarコマンドラインツール
- Pulsarコマンドラインツール
@@ -762,9 +762,9 @@
+ パーティションドトピック
- パーティションドトピック
@@ -792,9 +792,11 @@
+ メッセージの保存と有効期限
+
+
- メッセージの保存と有効期限
@@ -1143,6 +1145,8 @@
+
+
@@ -1387,6 +1391,8 @@
+
+