kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From danburk...@apache.org
Subject kudu git commit: docs: revise security doc based on KUDU-1875 and add release notes for 1.4
Date Tue, 23 May 2017 01:19:05 GMT
Repository: kudu
Updated Branches:
  refs/heads/master 952222b52 -> d354693c1


docs: revise security doc based on KUDU-1875 and
add release notes for 1.4

Change-Id: I2e386203f5ed3ef66e2ec136e67738b8c7eb8b1a
Reviewed-on: http://gerrit.cloudera.org:8080/6922
Reviewed-by: Dan Burkert <danburkert@apache.org>
Tested-by: Dan Burkert <danburkert@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/d354693c
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/d354693c
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/d354693c

Branch: refs/heads/master
Commit: d354693c17b802fa34ef9362a8bf9e748b887a6b
Parents: 952222b
Author: hahao <hao.hao@cloudera.com>
Authored: Thu May 18 12:20:02 2017 -0700
Committer: Dan Burkert <danburkert@apache.org>
Committed: Tue May 23 01:18:47 2017 +0000

----------------------------------------------------------------------
 docs/prior_release_notes.adoc | 174 +++++++++++++++++++++++++++++++++++
 docs/release_notes.adoc       | 179 +++++--------------------------------
 docs/security.adoc            |  23 +++--
 3 files changed, 212 insertions(+), 164 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/d354693c/docs/prior_release_notes.adoc
----------------------------------------------------------------------
diff --git a/docs/prior_release_notes.adoc b/docs/prior_release_notes.adoc
index ba1971e..f1ea542 100644
--- a/docs/prior_release_notes.adoc
+++ b/docs/prior_release_notes.adoc
@@ -37,6 +37,180 @@ reproduced on this page. Please consult the
 link:http://kudu.apache.org/releases/[documentation of the appropriate release]
 for a list of known issues and limitations.
 
+[[rn_1.3.0]]
+== Release notes specific to 1.3.0
+
+[[rn_1.3.0_new_features]]
+== New features
+
+* Kudu 1.3 adds support for strong authentication based on Kerberos. This optional feature
+  allows users to authenticate themselves using Kerberos tickets, and also provides
+  mutual authentication of servers using Kerberos credentials stored in keytabs. This
+  feature is optional, but recommended for deployments requiring security.
+
+* Kudu 1.3 adds support for encryption of data on the network using Transport Layer Security
+  (TLS). Kudu will now use TLS to encrypt all network traffic between clients and servers
as
+  well as any internal traffic among servers, with the exception of traffic determined to
+  be within a localhost network connection. Encryption is enabled by default whenever it
can
+  be determined that both the client and server support the feature.
+
+* Kudu 1.3 adds coarse-grained service-level authorization of access to the cluster.
+  The operator may set up lists of permitted users who may act as administrators and
+  as clients of the cluster. Combined with the strong authentication feature described
+  above, this can enable a secure environment for some use cases. Note that fine-grained
+  access control (e.g. table-level or column-level) is not yet supported.
+
+* Kudu 1.3 adds a background task to tablet servers which removes historical versions of
+  data which have fallen behind the configured data retention time. This reduces disk space
+  usage in all workloads, but particularly in those with a higher volume of updates or
+  upserts.
+
+* Kudu now incorporates Google Breakpad, a library which writes crash reports in
+  the case of a server crash. These reports can be found within the configured log directory,
+  and can be useful during bug diagnosis.
+
+
+== Optimizations and improvements
+
+* Kudu servers will now change the file permissions of data directories and contained
+  data files based on a new configuration flag `--umask`. As a result, after upgrading,
+  permissions on disk may be more restrictive than in previous versions. The new default
+  configuration improves data security.
+
+* Kudu's web UI will now redact strings which may include sensitive user data. For example,
+  the monitoring page which shows in-progress scans no longer includes the scanner predicate
+  values. The tracing and RPC diagnostics endpoints no longer include contents of RPCs which
+  may include table data.
+
+* By default, Kudu now reserves 1% of each configured data volume as free space. If a volume
+  is seen to have less than 1% of disk space free, Kudu will stop writing to that volume
+  to avoid completely filling up the disk.
+
+* The default encoding for numeric columns (int, float, and double) has been changed
+  to `BIT_SHUFFLE`. The default encoding for binary and string columns has been
+  changed to `DICT_ENCODING`. Dictionary encoding automatically falls back to the old
+  default (`PLAIN`) when cardinality is too high to be effectively encoded.
++
+These new defaults match the default behavior of other storage mechanisms such as
+  Apache Parquet and are likely to perform better out of the box.
+
+* Kudu now uses `LZ4` compression when writing its Write Ahead Log (WAL). This improves
+  write performance and stability for many use cases.
+
+* Kudu now uses `LZ4` compression when writing delta files. This can improve both
+  read and write performance as well as save substantial disk usage, especially
+  for workloads involving a high number of updates or upserts containing compressible
+  data.
+
+* The Kudu API now supports the ability to express `IS NULL` and `IS NOT NULL` predicates
+  on scanners. The Spark DataSource integration will take advantage of these new
+  predicates when possible.
+
+* Both {cpp} and Java clients have been optimized to prune partitions more effectively
+  when performing scans using the `IN (...)` predicate.
+
+* The exception messages produced by the Java client are now truncated to a maximum length
+  of 32KB.
+
+
+[[rn_1.3.0_fixed_issues]]
+== Fixed Issues
+
+* link:https://issues.apache.org/jira/browse/KUDU-1893[KUDU-1893]
+  Fixed a critical bug in which wrong results would be returned when evaluating
+  predicates applied to columns added using the `ALTER TABLE` operation.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1905[KUDU-1905]
+  Fixed a crash after inserting a row sharing a primary key with a recently-deleted
+  row in tables where the primary key is comprised of all of the columns.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1899[KUDU-1899]
+  Fixed a crash after inserting a row with an empty string as the single-column
+  primary key.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1904[KUDU-1904]
+  Fixed a potential crash when performing random reads against a column using RLE
+  encoding and containing long runs of NULL values.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1853[KUDU-1853]
+  Fixed an issue where disk space could be leaked on servers which experienced an error
+  during the process of copying tablet data from another server.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1856[KUDU-1856]
+  Fixed an issue in which disk space could be leaked by Kudu servers storing data on
+  partitions using the XFS file system. Any leaked disk space will be automatically
+  recovered upon upgrade.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1888[KUDU-1888],
+  link:https://issues.apache.org/jira/browse/KUDU-1906[KUDU-1906]
+  Fixed multiple issues in the Java client where operation callbacks would never be
+  triggered, causing the client to hang.
+
+
+[[rn_1.3.0_wire_compatibility]]
+== Wire Protocol compatibility
+
+Kudu 1.3.0 is wire-compatible with previous versions of Kudu:
+
+* Kudu 1.3 clients may connect to servers running Kudu 1.0. If the client uses features
+  that are not available on the target server, an error will be returned.
+* Kudu 1.0 clients may connect to servers running Kudu 1.3 with the exception of the
+  below-mentioned restrictions regarding secure clusters.
+* Rolling upgrade between Kudu 1.2 and Kudu 1.3 servers is believed to be possible
+  though has not been sufficiently tested. Users are encouraged to shut down all nodes
+  in the cluster, upgrade the software, and then restart the daemons on the new version.
+
+The authentication features newly introduced in Kudu 1.3 place the following limitations
+on wire compatibility with older versions:
+
+* If a Kudu 1.3 cluster is configured with authentication or encryption set to "required",
+  older clients will be unable to connect.
+* If a Kudu 1.3 cluster is configured with authentication and encryption set to "optional"
+  or "disabled", older clients will still be able to connect.
+
+
+[[rn_1.3.0_incompatible_changes]]
+== Incompatible Changes in Kudu 1.3.0
+
+* Due to storage format changes in Kudu 1.3, downgrade from Kudu 1.3 to earlier versions
+  is not supported. After upgrading to Kudu 1.3, attempting to restart with an earlier
+  version will result in an error.
+
+* In order to support running MapReduce and Spark jobs on secure clusters, these
+  frameworks now connect to the cluster at job submission time to retrieve authentication
+  credentials which can later be used by the tasks to be spawned. This means that
+  the process submitting jobs to Kudu clusters must have direct access to that cluster.
+
+* The embedded web servers in Kudu processes now specify the `X-Frame-Options: DENY` HTTP
+  header which prevents embedding Kudu web pages in HTML `iframe` elements.
+
+[[rn_1.3.0_client_compatibility]]
+=== Client Library Compatibility
+
+* The Kudu 1.3 Java client library is API- and ABI-compatible with Kudu 1.2. Applications
+  written against Kudu 1.2 will compile and run against the Kudu 1.3 client library and
+  vice-versa, unless one of the following newly added APIs is used:
+** `[Async]KuduClient.exportAuthenticationCredentials(...)` (unstable API)
+** `[Async]KuduClient.importAuthenticationCredentials(...)` (unstable API)
+** `[Async]KuduClient.getMasterAddressesAsString()`
+** `KuduPredicate.newIsNotNullPredicate()`
+** `KuduPredicate.newIsNullPredicate()`
+
+* The Kudu 1.3 {cpp} client is API- and ABI-forward-compatible with Kudu 1.2.
+  Applications written and compiled against the Kudu 1.2 client library will run without
+  modification against the Kudu 1.3 client library. Applications written and compiled
+  against the Kudu 1.3 client library will run without modification against the Kudu 1.2
+  client library unless they use one of the following new APIs:
+** `kudu::DisableOpenSSLInitialization()`
+** `KuduClientBuilder::import_authentication_credentials(...)`
+** `KuduClient::ExportAuthenticationCredentials(...)`
+** `KuduClient::NewIsNotNullPredicate(...)`
+** `KuduClient::NewIsNullPredicate(...)`
+
+* The Kudu 1.3 Python client is API-compatible with Kudu 1.2. Applications
+  written against Kudu 1.2 will continue to run against the Kudu 1.3 client
+  and vice-versa.
+
 
 [[rn_1.2.0]]
 == Release notes specific to 1.2.0

http://git-wip-us.apache.org/repos/asf/kudu/blob/d354693c/docs/release_notes.adoc
----------------------------------------------------------------------
diff --git a/docs/release_notes.adoc b/docs/release_notes.adoc
index bba1c4f..ee19f36 100644
--- a/docs/release_notes.adoc
+++ b/docs/release_notes.adoc
@@ -16,7 +16,7 @@
 // under the License.
 
 [[release_notes]]
-= Apache Kudu 1.3.0 Release Notes
+= Apache Kudu 1.4.0 Release Notes
 
 :author: Kudu Team
 :imagesdir: ./images
@@ -28,181 +28,44 @@
 :sectlinks:
 :experimental:
 
-[[rn_1.3.0]]
+[[rn_1.4.0]]
 
-[[rn_1.3.0_new_features]]
+[[rn_1.4.0_new_features]]
 == New features
 
-* Kudu 1.3 adds support for strong authentication based on Kerberos. This optional feature
-  allows users to authenticate themselves using Kerberos tickets, and also provides
-  mutual authentication of servers using Kerberos credentials stored in keytabs. This
-  feature is optional, but recommended for deployments requiring security.
-
-* Kudu 1.3 adds support for encryption of data on the network using Transport Layer Security
-  (TLS). Kudu will now use TLS to encrypt all network traffic between clients and servers
as
-  well as any internal traffic among servers, with the exception of traffic determined to
-  be within a localhost network connection. Encryption is enabled by default whenever it
can
-  be determined that both the client and server support the feature.
-
-* Kudu 1.3 adds coarse-grained service-level authorization of access to the cluster.
-  The operator may set up lists of permitted users who may act as administrators and
-  as clients of the cluster. Combined with the strong authentication feature described
-  above, this can enable a secure environment for some use cases. Note that fine-grained
-  access control (e.g. table-level or column-level) is not yet supported.
-
-* Kudu 1.3 adds a background task to tablet servers which removes historical versions of
-  data which have fallen behind the configured data retention time. This reduces disk space
-  usage in all workloads, but particularly in those with a higher volume of updates or
-  upserts.
-
-* Kudu now incorporates Google Breakpad, a library which writes crash reports in
-  the case of a server crash. These reports can be found within the configured log directory,
-  and can be useful during bug diagnosis.
-
 
 == Optimizations and improvements
 
-* Kudu servers will now change the file permissions of data directories and contained
-  data files based on a new configuration flag `--umask`. As a result, after upgrading,
-  permissions on disk may be more restrictive than in previous versions. The new default
-  configuration improves data security.
-
-* Kudu's web UI will now redact strings which may include sensitive user data. For example,
-  the monitoring page which shows in-progress scans no longer includes the scanner predicate
-  values. The tracing and RPC diagnostics endpoints no longer include contents of RPCs which
-  may include table data.
-
-* By default, Kudu now reserves 1% of each configured data volume as free space. If a volume
-  is seen to have less than 1% of disk space free, Kudu will stop writing to that volume
-  to avoid completely filling up the disk.
-
-* The default encoding for numeric columns (int, float, and double) has been changed
-  to `BIT_SHUFFLE`. The default encoding for binary and string columns has been
-  changed to `DICT_ENCODING`. Dictionary encoding automatically falls back to the old
-  default (`PLAIN`) when cardinality is too high to be effectively encoded.
+* Kudu servers, by default, will now only allow unencrypted or unauthenticated connections
+  from trusted subnets, which are private networks (127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,
+  192.168.0.0/16,169.254.0.0/16) and local subnets of all local network interfaces.
+  Unencrypted or unauthenticated connections from publicly routable IPs will be rejected,
+  even if encryption and authentication are not configured.
 +
-These new defaults match the default behavior of other storage mechanisms such as
-  Apache Parquet and are likely to perform better out of the box.
-
-* Kudu now uses `LZ4` compression when writing its Write Ahead Log (WAL). This improves
-  write performance and stability for many use cases.
-
-* Kudu now uses `LZ4` compression when writing delta files. This can improve both
-  read and write performance as well as save substantial disk usage, especially
-  for workloads involving a high number of updates or upserts containing compressible
-  data.
-
-* The Kudu API now supports the ability to express `IS NULL` and `IS NOT NULL` predicates
-  on scanners. The Spark DataSource integration will take advantage of these new
-  predicates when possible.
-
-* Both {cpp} and Java clients have been optimized to prune partitions more effectively
-  when performing scans using the `IN (...)` predicate.
-
-* The exception messages produced by the Java client are now truncated to a maximum length
-  of 32KB.
-
-[[rn_1.3.0_fixed_issues]]
+The trusted subnets can be configured using the `--trusted_subnets` flag, which can be set
+   to IP blocks represented in CIDR notation separated by comma. Set it to '0.0.0.0/0' to
+   allow unauthenticated connections from all remote IP addresses. However, if network access
+   is not otherwise restricted by a firewall, malicious users may be able to gain unauthorized
+   access. This can be mitigated if authentication and encryption are configured to be
+   required.
+
+[[rn_1.4.0_fixed_issues]]
 == Fixed Issues
 
 
-* link:https://issues.apache.org/jira/browse/KUDU-1893[KUDU-1893]
-  Fixed a critical bug in which wrong results would be returned when evaluating
-  predicates applied to columns added using the `ALTER TABLE` operation.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1905[KUDU-1905]
-  Fixed a crash after inserting a row sharing a primary key with a recently-deleted
-  row in tables where the primary key is comprised of all of the columns.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1899[KUDU-1899]
-  Fixed a crash after inserting a row with an empty string as the single-column
-  primary key.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1904[KUDU-1904]
-  Fixed a potential crash when performing random reads against a column using RLE
-  encoding and containing long runs of NULL values.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1853[KUDU-1853]
-  Fixed an issue where disk space could be leaked on servers which experienced an error
-  during the process of copying tablet data from another server.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1856[KUDU-1856]
-  Fixed an issue in which disk space could be leaked by Kudu servers storing data on
-  partitions using the XFS file system. Any leaked disk space will be automatically
-  recovered upon upgrade.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1888[KUDU-1888],
-  link:https://issues.apache.org/jira/browse/KUDU-1906[KUDU-1906]
-  Fixed multiple issues in the Java client where operation callbacks would never be
-  triggered, causing the client to hang.
-
-
-[[rn_1.3.0_wire_compatibility]]
+[[rn_1.4.0_wire_compatibility]]
 == Wire Protocol compatibility
 
-Kudu 1.3.0 is wire-compatible with previous versions of Kudu:
 
-* Kudu 1.3 clients may connect to servers running Kudu 1.0. If the client uses features
-  that are not available on the target server, an error will be returned.
-* Kudu 1.0 clients may connect to servers running Kudu 1.3 with the exception of the
-  below-mentioned restrictions regarding secure clusters.
-* Rolling upgrade between Kudu 1.2 and Kudu 1.3 servers is believed to be possible
-  though has not been sufficiently tested. Users are encouraged to shut down all nodes
-  in the cluster, upgrade the software, and then restart the daemons on the new version.
-
-The authentication features newly introduced in Kudu 1.3 place the following limitations
-on wire compatibility with older versions:
-
-* If a Kudu 1.3 cluster is configured with authentication or encryption set to "required",
-  older clients will be unable to connect.
-* If a Kudu 1.3 cluster is configured with authentication and encryption set to "optional"
-  or "disabled", older clients will still be able to connect.
-
-
-[[rn_1.3.0_incompatible_changes]]
+[[rn_1.4.0_incompatible_changes]]
 == Incompatible Changes in Kudu 1.3.0
 
-* Due to storage format changes in Kudu 1.3, downgrade from Kudu 1.3 to earlier versions
-  is not supported. After upgrading to Kudu 1.3, attempting to restart with an earlier
-  version will result in an error.
 
-* In order to support running MapReduce and Spark jobs on secure clusters, these
-  frameworks now connect to the cluster at job submission time to retrieve authentication
-  credentials which can later be used by the tasks to be spawned. This means that
-  the process submitting jobs to Kudu clusters must have direct access to that cluster.
-
-* The embedded web servers in Kudu processes now specify the `X-Frame-Options: DENY` HTTP
-  header which prevents embedding Kudu web pages in HTML `iframe` elements.
-
-[[rn_1.3.0_client_compatibility]]
+[[rn_1.4.0_client_compatibility]]
 === Client Library Compatibility
 
-* The Kudu 1.3 Java client library is API- and ABI-compatible with Kudu 1.2. Applications
-  written against Kudu 1.2 will compile and run against the Kudu 1.3 client library and
-  vice-versa, unless one of the following newly added APIs is used:
-** `[Async]KuduClient.exportAuthenticationCredentials(...)` (unstable API)
-** `[Async]KuduClient.importAuthenticationCredentials(...)` (unstable API)
-** `[Async]KuduClient.getMasterAddressesAsString()`
-** `KuduPredicate.newIsNotNullPredicate()`
-** `KuduPredicate.newIsNullPredicate()`
-
-* The Kudu 1.3 {cpp} client is API- and ABI-forward-compatible with Kudu 1.2.
-  Applications written and compiled against the Kudu 1.2 client library will run without
-  modification against the Kudu 1.3 client library. Applications written and compiled
-  against the Kudu 1.3 client library will run without modification against the Kudu 1.2
-  client library unless they use one of the following new APIs:
-** `kudu::DisableOpenSSLInitialization()`
-** `KuduClientBuilder::import_authentication_credentials(...)`
-** `KuduClient::ExportAuthenticationCredentials(...)`
-** `KuduClient::NewIsNotNullPredicate(...)`
-** `KuduClient::NewIsNullPredicate(...)`
-
-* The Kudu 1.3 Python client is API-compatible with Kudu 1.2. Applications
-  written against Kudu 1.2 will continue to run against the Kudu 1.3 client
-  and vice-versa.
-
-
-[[rn_1.3.0_known_issues]]
+
+[[rn_1.4.0_known_issues]]
 
 == Known Issues and Limitations
 

http://git-wip-us.apache.org/repos/asf/kudu/blob/d354693c/docs/security.adoc
----------------------------------------------------------------------
diff --git a/docs/security.adoc b/docs/security.adoc
index 1b8cbf7..d55a347 100644
--- a/docs/security.adoc
+++ b/docs/security.adoc
@@ -46,9 +46,19 @@ Authentication can be configured on Kudu servers using the
 `--rpc-authentication` flag, which can be set to `required`, `optional`, or
 `disabled`. By default, the flag is set to `optional`. When `required`, Kudu
 will reject connections from clients and servers who lack authentication
-credentials. When `optional`, Kudu will attempt to use strong authentication,
-but will allow unauthenticated connections. When `disabled`, Kudu will only
-allow unauthenticated connections.
+credentials. When `optional`, Kudu will attempt to use strong authentication.
+When `disabled` or strong authentication fails for 'optional', by default Kudu
+will only allow unauthenticated connections from trusted subnets, which are
+private networks (127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,
+169.254.0.0/16) and local subnets of all local network interfaces. Unauthenticated
+connections from publicly routable IPs will be rejected.
+
+The trusted subnets can be configured using the `--trusted_subnets` flag,
+which can be set to IP blocks in CIDR notation separated by comma. Set it to
+'0.0.0.0/0' to allow unauthenticated connections from all remote IP addresses.
+However, if network access is not otherwise restricted by a firewall,
+malicious users may be able to gain unauthorized access. This can be mitigated
+if authentication is configured to be required.
 
 WARNING: When the `--rpc-authentication` flag is set to `optional`,
 the cluster does not prevent access from unauthenticated users. To secure a
@@ -106,9 +116,10 @@ to be encrypted with TLS.
 Encryption can be configured on Kudu servers using the `--rpc-encryption` flag,
 which can be set to `required`, `optional`, or `disabled`. By default, the flag
 is set to `optional`. When `required`, Kudu will reject unencrypted connections.
-When `optional`, Kudu will attempt to use encryption, but will allow unencrypted
-connections. When `disabled`, Kudu will never use encryption. To secure a
-cluster, use `--rpc-encryption=required`.
+When `optional`, Kudu will attempt to use encryption. Same as authentication,
+when `disabled` or encryption fails for `optional`, Kudu will only allow
+unencrypted connections from trusted subnets and reject any unencrypted connections
+from publicly routable IPs. To secure a cluster, use `--rpc-encryption=required`.
 
 NOTE: Kudu will automatically turn off encryption on local loopback connections,
 since traffic from these connections is never exposed externally. This allows


Mime
View raw message