kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From granthe...@apache.org
Subject [41/51] [partial] kudu git commit: Update website for 1.7.0 release
Date Wed, 21 Mar 2018 19:40:34 GMT
http://git-wip-us.apache.org/repos/asf/kudu/blob/d78c4caa/docs/release_notes.html
----------------------------------------------------------------------
diff --git a/docs/release_notes.html b/docs/release_notes.html
index e4d2e83..5d6c96f 100644
--- a/docs/release_notes.html
+++ b/docs/release_notes.html
@@ -1,8 +1,8 @@
 ---
-title: Apache Kudu 1.6.0 Release Notes
+title: Apache Kudu 1.7.0 Release Notes
 layout: default
 active_nav: docs
-last_updated: 'Last updated 2017-12-01 23:56:01 PST'
+last_updated: 'Last updated 2018-03-19 08:09:15 CDT'
 ---
 <!--
 
@@ -24,16 +24,21 @@ limitations under the License.
   <div class="row">
     <div class="col-md-9">
 
-<h1>Apache Kudu 1.6.0 Release Notes</h1>
+<h1>Apache Kudu 1.7.0 Release Notes</h1>
       <div class="sect1">
-<h2 id="rn_1.6.0_upgrade_notes"><a class="link" href="#rn_1.6.0_upgrade_notes">Upgrade Notes</a></h2>
+<h2 id="rn_1.7.0_upgrade_notes"><a class="link" href="#rn_1.7.0_upgrade_notes">Upgrade Notes</a></h2>
 <div class="sectionbody">
 <div class="ulist">
 <ul>
 <li>
-<p>Upgrading directly from Kudu 1.5.0 is supported and no special upgrade steps
-are required. A rolling upgrade may work, however it has not been tested.
-When upgrading Kudu, it is recommended to first shut down all Kudu processes
+<p>Upgrading directly from Kudu 1.6.0 is supported and no special upgrade steps
+are required. A rolling upgrade of the server side will <em>not</em> work because
+the default replica management scheme changed, and running masters and tablet
+servers with different replica management schemes is not supported, see
+<a href="#rn_1.7.0_incompatible_changes">Incompatible Changes in Kudu 1.7.0</a> for details. However, mixing client and
+server sides of different versions is not a problem. You can still
+update your clients before your servers or vice versa.
+When upgrading to Kudu 1.7, it is required to first shut down all Kudu processes
 across the cluster, then upgrade the software on all servers, then restart
 the Kudu processes on all servers in the cluster.</p>
 </li>
@@ -42,20 +47,20 @@ the Kudu processes on all servers in the cluster.</p>
 </div>
 </div>
 <div class="sect1">
-<h2 id="rn_1.6.0_obsoletions"><a class="link" href="#rn_1.6.0_obsoletions">Obsoletions</a></h2>
+<h2 id="rn_1.7.0_obsoletions"><a class="link" href="#rn_1.7.0_obsoletions">Obsoletions</a></h2>
 <div class="sectionbody">
 <div class="ulist">
 <ul>
 <li>
-<p>Support for Spark 1 (kudu-spark_2.10) has been <strong>removed</strong> in Kudu 1.6.0 and
-now only Spark 2 is supported. Spark 1 support was deprecated in Kudu 1.5.0.</p>
+<p>The <code>tcmalloc_contention_time</code> metric, which previously tracked the amount
+of time spent in memory allocator lock contention, has been removed.</p>
 </li>
 </ul>
 </div>
 </div>
 </div>
 <div class="sect1">
-<h2 id="rn_1.6.0_deprecations"><a class="link" href="#rn_1.6.0_deprecations">Deprecations</a></h2>
+<h2 id="rn_1.7.0_deprecations"><a class="link" href="#rn_1.7.0_deprecations">Deprecations</a></h2>
 <div class="sectionbody">
 <div class="ulist">
 <ul>
@@ -68,37 +73,67 @@ the next major release.</p>
 </div>
 </div>
 <div class="sect1">
-<h2 id="rn_1.6.0_new_features"><a class="link" href="#rn_1.6.0_new_features">New features</a></h2>
+<h2 id="rn_1.7.0_new_features"><a class="link" href="#rn_1.7.0_new_features">New features</a></h2>
 <div class="sectionbody">
 <div class="ulist">
 <ul>
 <li>
-<p>Tablet servers' tolerance of disk failures is now enabled by default and has
-been extended to handle data directory failures at runtime. In the event of
-a disk failure at runtime, any tablets with data on a failed disk will be
-shut down and restarted on another tablet server. There is a configurable
-tradeoff between a newly added tablet&#8217;s tolerance to disk failures and its
-ability to parallelize reads via the experimental
-<code>--fs_target_data_dirs_per_tablet</code> flag. Tablets that are spread across fewer
-disks are less likely to be affected by a disk failure, at the cost of
-reduced parallelism. By default, tablets are striped across all available
-disks. Note that the first configured data directory and the WAL directory
-cannot currently tolerate disk failures. This will be further improved in
-future Kudu releases.</p>
-</li>
-<li>
-<p>Kudu servers can now adopt new data directories via the new
-<code>kudu fs update_dirs</code> tool. The new directory will be used by new tablet
-replicas only. Note that removing directories is not yet supported
-(see <a href="https://issues.apache.org/jira/browse/KUDU-2202">KUDU-2202</a>).</p>
-</li>
-<li>
-<p>Kudu servers have two new flags to control webui TLS/HTTPS
-settings: <code>--webserver_tls_ciphers</code> and <code>--webserver_tls_min_protocol</code>.
-These flags allow the advertised TLS ciphers and TLS protocol versions to be
-configured. Additionally, the webserver now excludes insecure legacy ciphers
-by default
-(see <a href="https://issues.apache.org/jira/browse/KUDU-2190">KUDU-2190</a>).</p>
+<p>Kudu now supports the decimal column type. The decimal type is a numeric data type
+with fixed scale and precision suitable for financial and other arithmetic
+calculations where the imprecise representation and rounding behavior of float and
+double make those types impractical. The decimal type is also useful for integers
+larger than int64 and cases with fractional values in a primary key.
+See <a href="schema_design.html#decimal">Decimal Type</a> for more details.</p>
+</li>
+<li>
+<p>The strategy Kudu uses for automatically healing tablets which have lost a
+replica due to server or disk failures has been improved. The new re-replication
+strategy, or replica management scheme, first adds a replacement tablet replica
+before evicting the failed one. With the previous replica management scheme,
+the system first evicts the failed replica and then adds a replacement. The new
+replica management scheme allows for much faster recovery of tablets in
+scenarios where one tablet server goes down and then returns back shortly after
+5 minutes or so. The new scheme also provides substantially better overall
+stability on clusters with frequent server failures.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-1097">KUDU-1097</a>).</p>
+</li>
+<li>
+<p>The <code>kudu fs update_dirs</code> tool now supports removing directories. Unless the
+<code>--force</code> flag is specified, Kudu will not allow the removal of a directory
+across which tablets are configured to spread data. If specified, all tablet
+replicas configured to use that directory will fail upon starting up and be
+replicated elsewhere, provided a majority exists elsewhere.</p>
+</li>
+<li>
+<p>Users can use the new <code>--fs_metadata_dir</code> to specify the directory in which
+to place tablet-specific metadata. It is recommended, although not
+necessary, that this be placed on a high-performance drive with high
+bandwidth and low latency, e.g. a solid-state drive. If not specified,
+metadata will be placed in the directory specified by <code>--fs_wal_dir</code>, or in
+the directory specified by the first entry of <code>--fs_data_dirs</code> if metadata
+already exists there from a pre-Kudu 1.7 deployment. Kudu will not
+automatically move existing metadata based on this configuration.</p>
+</li>
+<li>
+<p>Kudu 1.7 introduces a new scan read mode READ_YOUR_WRITES. Users can specify
+READ_YOUR_WRITES when creating a new scanner in C++, Java and Python clients.
+If this mode is used, the client will perform a read such that it follows all
+previously known writes and reads from this client. Reads in this mode ensure
+read-your-writes and read-your-reads session guarantees, while minimizing
+latency caused by waiting for outstanding write transactions to complete.
+Note that this is still an experimental feature which may be stabilized in
+future releases.</p>
+</li>
+<li>
+<p>The tablet server web UI scans dashboard (/scans) has been improved with
+several new features, including: showing the most recently completed scans,
+a pseudo-SQL scan descriptor that concisely shows the selected columns and
+applied predicates, and more complete and better documented scan statistics.</p>
+</li>
+<li>
+<p>Kudu daemons now expose a web page <code>/stacks</code> which dumps the current stack
+trace of every thread running in the server. This information can be helpful
+when diagnosing performance issues.</p>
 </li>
 </ul>
 </div>
@@ -110,161 +145,209 @@ by default
 <div class="ulist">
 <ul>
 <li>
-<p>Kudu servers can now tolerate short interruptions in NTP clock
-synchronization. NTP synchronization is still required when any Kudu daemon
-starts up. If NTP synchronization is not available, diagnostic information
-is now logged to help pinpoint the issue
-(see <a href="https://issues.apache.org/jira/browse/KUDU-1578">KUDU-1578</a>).</p>
-</li>
-<li>
-<p>Tablet server startup time has been improved significantly on servers
-containing large numbers of blocks.</p>
-</li>
-<li>
-<p>The log block manager now performs disk data deletion in batches. This
-optimization can significantly reduce the time taken to delete data on a tablet.</p>
+<p>By default, each tablet replica will now stripe data blocks across 3 data
+directories instead of all data directories. This decreases the likelihood
+that any given tablet will be affected in the event of a single disk failure.
+No substantial performance impact is expected due to this feature based on
+<a href="https://github.com/apache/kudu/commit/60276c54a221d554287c6645df7df542fe6d6443">performance testing</a>.
+This change only affects new replicas created after upgrading to Kudu 1.7.</p>
 </li>
 <li>
-<p>The usage of sensitive data redaction flag has been slightly changed. By
-setting <code>--redact=log</code> flag, redaction will be disabled in the web UI but
-retained for server logs. Alternatively, <code>--redact=none</code> can be used to
-disable redaction completely.</p>
+<p>Kudu servers previously offered the ability to enable a separate metrics log
+which stores periodic snapshots of all metrics available on a server. This
+functionality is now available as part of a more general “diagnostics log”
+which is enabled by default. The diagnostics log includes periodic dumps of
+server metrics as well as collections of thread stack traces. The default
+configuration ensures that no more than 640MB of diagnostics logs are retained,
+and typically the space consumption is significantly less due to compression.
+The format and contents of this log file are documented in the
+<a href="administration.html">Administration guide</a>.</p>
 </li>
 <li>
-<p>The Spark DataSource integration now can take advantage of scan locality
-for better scan performance, the scan will take place at the closest replica
-instead of going to the leader.</p>
+<p>The handling of errors in the synchronous Java client has been improved so that,
+when an exception is thrown, the stack trace indicates the correct location
+where the client function was invoked rather than a call stack of an internal
+worker thread. The original call stack from the worker thread is available as
+a “suppressed exception”.</p>
 </li>
 <li>
-<p>Various optimizations were made to reduce the 99th percentile latency of
-writes on the tablet server. This can also improve throughput on certain
-write workloads, particularly on larger clusters.</p>
+<p>The logging of errors in the Java client has been improved to exclude exception
+stack traces for expected scenarios such as failure to connect to a server in a
+cluster. Instead, only a single line informational message will be logged in
+such cases to aid in debugging.</p>
 </li>
 <li>
-<p>Kudu may now be configured to ignore system-wide auth_to_local mappings
-configured in /etc/krb5.conf by setting the configuration flag
-<code>--use_system_auth_to_local=false</code>
-(see <a href="https://issues.apache.org/jira/browse/KUDU-2198">KUDU-2198</a>).</p>
+<p>The Java client now uses a predefined prioritized list of TLS ciphers when
+establishing an encrypted connection to Kudu servers. This cipher list matches
+the list of ciphers preferred for server-to-server communication and ensures
+that the most efficient and secure ciphers are preferred. When the Kudu client
+is running on Java 8 or newer, this provides a substantial speed-up to read
+and write performance.</p>
 </li>
 <li>
-<p>The performance of the compaction scheduler has been improved. In
-previous versions, certain types of time series workloads were found to
-cause compaction scheduling to take tens of seconds. These workloads now
-schedule compactions an order of magnitude more efficiently.</p>
+<p>Reporting for the <code>kudu cluster ksck</code> tool has been updated so tablets and
+tables with on-going tablet copies are shown as "recovering". Additional
+reporting changes have been made to make various common scenarios,
+particularly tablet copies, less alarming.</p>
 </li>
 <li>
-<p>The compaction scheduler has been improved to avoid running a compaction
-when the benefit of that compaction is extremely small.</p>
+<p>The performance of inserting rows containing many string or binary columns has
+been improved, especially in the case of highly concurrent write workloads.</p>
 </li>
 <li>
-<p>Tablet servers now consider the health of all replicas of a tablet before
-deciding to evict one. This can improve stability of the Kudu cluster after
-experiencing multiple simultaneous daemon failures
-(see <a href="https://issues.apache.org/jira/browse/KUDU-2048">KUDU-2048</a>).</p>
+<p>By default, Spark tasks that scan Kudu will now be able to scan non-leader
+replicas. This allows Spark to more easily schedule kudu-spark tasks local to
+the data. Users can disable this behavior by passing 'leader_only' to the
+'kudu.scanLocality' option."</p>
 </li>
 <li>
-<p>Several performance improvements have been made to the Kudu master,
-particularly in concurrency of clients opening tables. This should improve
-performance in highly concurrent workloads.</p>
+<p>The number of OS threads used in the steady state and during bursts of
+activity (such as in Raft leader elections triggered by a node failure) has
+been drastically reduced and should no longer exceed the value of <code>ulimit -u</code>.
+As such, it should no longer be necessary to increase the value of <code>ulimit -u</code>
+(or of /proc/sys/kernel/threads-max) in order to run a Kudu tablet server in
+most cases.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-1913">KUDU-1913</a>).</p>
 </li>
 <li>
-<p>The on-disk size metric for a tablet now includes all data and metadata.
-Previously, it excluded WAL segments and consensus metadata
-(see <a href="https://issues.apache.org/jira/browse/KUDU-1755">KUDU-1755</a>).</p>
+<p>An issue where sparse column predicates could cause excessive data-block reads
+has been fixed. Previously in certain scans with sparsely matching predicates
+on multiple columns, Kudu would read and decode the same data blocks many times.
+The improvement typically results in a 5-10x performance increase for the
+affected scans.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2231">KUDU-2231</a>).</p>
 </li>
 <li>
-<p>Added verbose mode for the 'kudu cluster ksck' command to enable output
-of detailed information on the cluster&#8217;s metadata, even when no errors are
-detected.</p>
+<p>The efficiency and on-disk size of large updated values has been improved.
+This will improve update-heavy workloads which overwrite large (1KiB+) values.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2253">KUDU-2253</a>).</p>
 </li>
 </ul>
 </div>
 </div>
 </div>
 <div class="sect1">
-<h2 id="rn_1.6.0_fixed_issues"><a class="link" href="#rn_1.6.0_fixed_issues">Fixed Issues</a></h2>
+<h2 id="rn_1.7.0_fixed_issues"><a class="link" href="#rn_1.7.0_fixed_issues">Fixed Issues</a></h2>
 <div class="sectionbody">
 <div class="ulist">
 <ul>
 <li>
-<p>HybridTime timestamp propagation now works in the Java client when using scan
-tokens (see <a href="https://issues.apache.org/jira/browse/KUDU-1411">KUDU-1411</a>).</p>
+<p>Fixed a scenario where the on-disk data of a tablet server was completely
+erased and and a new tablet server was started on the same host. This issue
+could prevent tablet replicas previously hosted on the server from being
+evicted and re-replicated.
+Tablets now immediately evict replicas that respond with a different server
+UUID than expected.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-1613">KUDU-1613</a>).</p>
 </li>
 <li>
-<p>Fixed an error message commonly found in tablet server logs indicating that
-operations were being read "from the future"
-(see <a href="https://issues.apache.org/jira/browse/KUDU-1078">KUDU-1078</a>).</p>
+<p>Fixed a rare race condition when connecting to masters during their
+startup which might cause a client to get a response without a CA certificate
+and/or authentication token. This would cause the client to fail to authenticate
+with other servers in the cluster. The leader master now always sends a CA
+certificate and an authentication token (when applicable) to a Kudu client
+with a successful ConnectToMaster response.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-1927">KUDU-1927</a>).</p>
 </li>
 <li>
-<p>Tombstoned tablets no longer report metrics
-(see <a href="https://issues.apache.org/jira/browse/KUDU-2044">KUDU-2044</a>).</p>
+<p>The Kudu Java client now will retry a connection if no master is discovered as a
+leader, and the user has a valid authentication token. This avoids failure
+in recoverable cases when masters are in the process of the very first leader
+election after starting up.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2262">KUDU-2262</a>).</p>
 </li>
 <li>
-<p>Fixed a bug in the C++ client which could cause tablets to be erroneously
-pruned, or skipped, during certain scans, resulting in fewer results than
-expected being returned from queries. The bug only affected tables whose range
-partition columns are a proper prefix of the primary key
-(see <a href="https://issues.apache.org/jira/browse/KUDU-2173">KUDU-2173</a>).</p>
+<p>The Java client will now automatically attempt to re-acquire Kerberos
+credentials from the ticket cache when the prior credentials are about to
+expire. This allows client instances to persist longer than the expiration
+time of a single Kerberos ticket so long as some other process renews the
+credentials in the ticket cache. Documentation on interacting with Kerberos
+authentication has been added to the Javadoc for the <code>AsyncKuduClient</code> class.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2264">KUDU-2264</a>).</p>
 </li>
 <li>
-<p>Published Kudu Java artifacts are now fully compatible with JRE 7 and JRE 8.
-There was previously a bug in the release process which made them compatible
-only with JRE 8
-(see <a href="https://issues.apache.org/jira/browse/KUDU-2188">KUDU-2188</a>).</p>
+<p>Follower masters are now able to verify authentication tokens even if they have never
+been a leader. Prior to this fix, if a follower master had never been a leader,
+clients would be unable to authenticate to that master, resulting in spurious
+error messages being logged.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2265">KUDU-2265</a>).</p>
 </li>
 <li>
-<p>Fixed a typo in the list of default TLS ciphers used by Kudu servers. As a
-result, two additional cipher suites are now available:</p>
-<div class="ulist">
-<ul>
+<p>Fixed a tablet server crash when a tablet replica is deleted during a scan.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2295">KUDU-2295</a>).</p>
+</li>
 <li>
-<p>ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(128)  Mac=SHA256</p>
+<p>The evaluation order of predicates in scans with multiple predicates has been
+made deterministic. Due to a bug, this was not necessarily the case previously.
+Predicates are applied in most to least selective order, with ties broken by
+column index. The evaluation order may change in the future, particularly when
+better column statistics are made available internally.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2312">KUDU-2312</a>).</p>
 </li>
 <li>
-<p>AES256-GCM-SHA384       TLSv1.2 Kx=RSA  Au=RSA Enc=AESGCM(256) Mac=AEAD</p>
+<p>Previously, the <code>kudu tablet change_config move_replica</code> tool required all
+tablet servers in the cluster to be available when performing a move. This
+restriction has been relaxed: only the tablet server that will receive a replica
+of the tablet being moved and the hosts of the tablet&#8217;s existing replicas need to be
+available for the move to occur.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2331">KUDU-2331</a>).</p>
 </li>
-</ul>
-</div>
+<li>
+<p>Fixed a bug in the Java client which prevented the client from locating the
+new leader master after a leader failover in the case that the previous leader
+either remained online or restarted quickly. This bug resulted in the client
+timing out operations with errors indicating that there was no leader master.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2343">KUDU-2343</a>).</p>
+</li>
+<li>
+<p>The Unix process username of the client is now included inside the exported
+security credentials, so that the effective username of clients who import
+credentials and subsequently use unauthenticated (SASL PLAIN) connections
+matches the client who exported the security credentials. For example, this is
+useful to let the Spark executors know which username to use if the Spark
+driver has no authentication token. This change only affects clusters with
+encryption disabled using <code>--rpc-encryption=disabled</code>.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2259">KUDU-2259</a>).</p>
 </li>
 </ul>
 </div>
 </div>
 </div>
 <div class="sect1">
-<h2 id="rn_1.6.0_wire_compatibility"><a class="link" href="#rn_1.6.0_wire_compatibility">Wire Protocol compatibility</a></h2>
+<h2 id="rn_1.7.0_wire_compatibility"><a class="link" href="#rn_1.7.0_wire_compatibility">Wire Protocol compatibility</a></h2>
 <div class="sectionbody">
 <div class="paragraph">
-<p>Kudu 1.6.0 is wire-compatible with previous versions of Kudu:</p>
+<p>Kudu 1.7.0 is wire-compatible with previous versions of Kudu:</p>
 </div>
 <div class="ulist">
 <ul>
 <li>
-<p>Kudu 1.6 clients may connect to servers running Kudu 1.0 or later. If the client uses
+<p>Kudu 1.7 clients may connect to servers running Kudu 1.0 or later. If the client uses
 features that are not available on the target server, an error will be returned.</p>
 </li>
 <li>
-<p>Rolling upgrade between Kudu 1.5 and Kudu 1.6 servers is believed to be possible
+<p>Rolling upgrade between Kudu 1.6 and Kudu 1.7 servers is believed to be possible
 though has not been sufficiently tested. Users are encouraged to shut down all nodes
 in the cluster, upgrade the software, and then restart the daemons on the new version.</p>
 </li>
 <li>
-<p>Kudu 1.0 clients may connect to servers running Kudu 1.6 with the exception of the
+<p>Kudu 1.0 clients may connect to servers running Kudu 1.7 with the exception of the
 below-mentioned restrictions regarding secure clusters.</p>
 </li>
 </ul>
 </div>
 <div class="paragraph">
 <p>The authentication features introduced in Kudu 1.3 place the following limitations
-on wire compatibility between Kudu 1.6 and versions earlier than 1.3:</p>
+on wire compatibility between Kudu 1.7 and versions earlier than 1.3:</p>
 </div>
 <div class="ulist">
 <ul>
 <li>
-<p>If a Kudu 1.6 cluster is configured with authentication or encryption set to "required",
+<p>If a Kudu 1.7 cluster is configured with authentication or encryption set to "required",
 clients older than Kudu 1.3 will be unable to connect.</p>
 </li>
 <li>
-<p>If a Kudu 1.6 cluster is configured with authentication and encryption set to "optional"
+<p>If a Kudu 1.7 cluster is configured with authentication and encryption set to "optional"
 or "disabled", older clients will still be able to connect.</p>
 </li>
 </ul>
@@ -272,36 +355,75 @@ or "disabled", older clients will still be able to connect.</p>
 </div>
 </div>
 <div class="sect1">
-<h2 id="rn_1.6.0_incompatible_changes"><a class="link" href="#rn_1.6.0_incompatible_changes">Incompatible Changes in Kudu 1.6.0</a></h2>
+<h2 id="rn_1.7.0_incompatible_changes"><a class="link" href="#rn_1.7.0_incompatible_changes">Incompatible Changes in Kudu 1.7.0</a></h2>
 <div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>The newly introduced replica management scheme is not compatible with the
+old scheme, so it&#8217;s not possible to run pre-1.7 Kudu masters with
+1.7 Kudu tablet servers or vice versa. This is a server-side
+incompatibility only and it does not affect client compatibility. In other words,
+Kudu clients of prior versions are compatible with upgraded Kudu clusters.</p>
+<div class="ulist">
+<ul>
+<li>
+<p>Kudu masters of 1.7 version will not register Kudu tablet servers of 1.6
+and prior versions.</p>
+</li>
+<li>
+<p>Kudu tablet servers of 1.7 version will not work with Kudu masters of 1.6
+and prior versions.</p>
+</li>
+</ul>
+</div>
+</li>
+<li>
+<p>The format of the previously-optional metrics log has changed to include a
+human-readable timestamp on each line. The path of the log file has also
+changed with the word “diagnostics” replacing the word “metrics” in the file
+name. The metrics log has been optimized to only include those metrics which
+have changed in between successive samples, and to not include entity attributes
+such as tablet partition information in the log.
+(see <a href="https://issues.apache.org/jira/browse/KUDU-2297">KUDU-2297</a>).</p>
+</li>
+</ul>
+</div>
 <div class="sect2">
-<h3 id="rn_1.6.0_client_compatibility"><a class="link" href="#rn_1.6.0_client_compatibility">Client Library Compatibility</a></h3>
+<h3 id="rn_1.7.0_client_compatibility"><a class="link" href="#rn_1.7.0_client_compatibility">Client Library Compatibility</a></h3>
 <div class="ulist">
 <ul>
 <li>
-<p>The Kudu 1.6 Java client library is API- and ABI-compatible with Kudu 1.5. Applications
-written against Kudu 1.5 will compile and run against the Kudu 1.6 client library and
+<p>The Kudu 1.7 Java client library is API- and ABI-compatible with Kudu 1.6. Applications
+written against Kudu 1.6 will compile and run against the Kudu 1.7 client library and
 vice-versa.</p>
 </li>
 <li>
-<p>The Kudu 1.6 C++ client is API- and ABI-forward-compatible with Kudu 1.5.
-Applications written and compiled against the Kudu 1.5 client library will run without
-modification against the Kudu 1.6 client library. Applications written and compiled
-against the Kudu 1.6 client library will run without modification against the Kudu 1.5
+<p>The Kudu 1.7 C++ client is API- and ABI-forward-compatible with Kudu 1.6.
+Applications written and compiled against the Kudu 1.6 client library will run without
+modification against the Kudu 1.7 client library. Applications written and compiled
+against the Kudu 1.7 client library will run without modification against the Kudu 1.6
 client library.</p>
 </li>
 <li>
-<p>The Kudu 1.6 Python client is API-compatible with Kudu 1.5. Applications
-written against Kudu 1.5 will continue to run against the Kudu 1.6 client
+<p>The Kudu 1.7 Python client is API-compatible with Kudu 1.6. Applications
+written against Kudu 1.6 will continue to run against the Kudu 1.7 client
 and vice-versa.</p>
 </li>
+<li>
+<p>Kudu 1.7 clients that attempt to create a table with a decimal column on a
+target server running Kudu 1.6 or earlier will receive an error response.
+Similarly Kudu clients running Kudu 1.6 or earlier will result in an error
+when attempting to access any table containing containing a decimal
+column.</p>
+</li>
 </ul>
 </div>
 </div>
 </div>
 </div>
 <div class="sect1">
-<h2 id="rn_1.6.0_known_issues"><a class="link" href="#rn_1.6.0_known_issues">Known Issues and Limitations</a></h2>
+<h2 id="rn_1.7.0_known_issues"><a class="link" href="#rn_1.7.0_known_issues">Known Issues and Limitations</a></h2>
 <div class="sectionbody">
 <div class="paragraph">
 <p>Please refer to the <a href="known_issues.html">Known Issues and Limitations</a> section of the
@@ -310,14 +432,14 @@ documentation.</p>
 </div>
 </div>
 <div class="sect1">
-<h2 id="rn_1.6.0_contributors"><a class="link" href="#rn_1.6.0_contributors">Contributors</a></h2>
+<h2 id="rn_1.7.0_contributors"><a class="link" href="#rn_1.7.0_contributors">Contributors</a></h2>
 <div class="sectionbody">
 <div class="paragraph">
-<p>Kudu 1.6 includes contributions from 14 people, including one
-first-time contributor, Hector Camarena.</p>
+<p>Kudu 1.7 includes contributions from 22 people, including two first-time
+contributors, Clemens Valiente and Tsuyoshi Ozawa.</p>
 </div>
 <div class="paragraph">
-<p>Thanks for helping to make Kudu even better!</p>
+<p>Thank you for helping to make Kudu even better!</p>
 </div>
 </div>
 </div>
@@ -381,20 +503,20 @@ first-time contributor, Hector Camarena.</p>
       <li>
 <span class="active-toc">Kudu Release Notes</span>
             <ul class="sectlevel1">
-<li><a href="#rn_1.6.0_upgrade_notes">Upgrade Notes</a></li>
-<li><a href="#rn_1.6.0_obsoletions">Obsoletions</a></li>
-<li><a href="#rn_1.6.0_deprecations">Deprecations</a></li>
-<li><a href="#rn_1.6.0_new_features">New features</a></li>
+<li><a href="#rn_1.7.0_upgrade_notes">Upgrade Notes</a></li>
+<li><a href="#rn_1.7.0_obsoletions">Obsoletions</a></li>
+<li><a href="#rn_1.7.0_deprecations">Deprecations</a></li>
+<li><a href="#rn_1.7.0_new_features">New features</a></li>
 <li><a href="#_optimizations_and_improvements">Optimizations and improvements</a></li>
-<li><a href="#rn_1.6.0_fixed_issues">Fixed Issues</a></li>
-<li><a href="#rn_1.6.0_wire_compatibility">Wire Protocol compatibility</a></li>
-<li><a href="#rn_1.6.0_incompatible_changes">Incompatible Changes in Kudu 1.6.0</a>
+<li><a href="#rn_1.7.0_fixed_issues">Fixed Issues</a></li>
+<li><a href="#rn_1.7.0_wire_compatibility">Wire Protocol compatibility</a></li>
+<li><a href="#rn_1.7.0_incompatible_changes">Incompatible Changes in Kudu 1.7.0</a>
 <ul class="sectlevel2">
-<li><a href="#rn_1.6.0_client_compatibility">Client Library Compatibility</a></li>
+<li><a href="#rn_1.7.0_client_compatibility">Client Library Compatibility</a></li>
 </ul>
 </li>
-<li><a href="#rn_1.6.0_known_issues">Known Issues and Limitations</a></li>
-<li><a href="#rn_1.6.0_contributors">Contributors</a></li>
+<li><a href="#rn_1.7.0_known_issues">Known Issues and Limitations</a></li>
+<li><a href="#rn_1.7.0_contributors">Contributors</a></li>
 <li><a href="#resources_and_next_steps">Resources</a></li>
 <li><a href="#_installation_options">Installation Options</a></li>
 <li><a href="#_next_steps">Next Steps</a></li>

http://git-wip-us.apache.org/repos/asf/kudu/blob/d78c4caa/docs/scaling_guide.html
----------------------------------------------------------------------
diff --git a/docs/scaling_guide.html b/docs/scaling_guide.html
new file mode 100644
index 0000000..b8d1a26
--- /dev/null
+++ b/docs/scaling_guide.html
@@ -0,0 +1,455 @@
+---
+title: Apache Kudu Scaling Guide
+layout: default
+active_nav: docs
+last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+---
+<!--
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+
+<div class="container">
+  <div class="row">
+    <div class="col-md-9">
+
+<h1>Apache Kudu Scaling Guide</h1>
+      <div id="preamble">
+<div class="sectionbody">
+<div class="paragraph">
+<p>This document describes in detail how Kudu scales with respect to various system resources,
+including memory, file descriptors, and threads. See the
+<a href="known_issues.html#_scale">scaling limits</a> for the maximum recommended parameters of a Kudu
+cluster. They can be used to estimate roughly the number of servers required for a given quantity
+of data.</p>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+The recommendations and conclusions here are only approximations. Appropriate numbers
+depend on use case. There is no substitute for measurement and monitoring of resources used during a
+representative workload.
+</td>
+</tr>
+</table>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_terms"><a class="link" href="#_terms">Terms</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>We will use the following terms:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><strong>hot replica</strong>: A tablet replica that is continuously receiving writes. For example, in a time
+series use case, tablet replicas for the most recent range partition on a time column would be
+continuously receiving the latest data, and would be hot replicas.</p>
+</li>
+<li>
+<p><strong>cold replica</strong>: A tablet replica that is not hot, i.e. a replica that is not frequently receiving
+writes, for example, once every few minutes. A cold replica may be read from. For example, in a time
+series use case, tablet replicas for previous range partitions on a time column would not receive
+writes at all, or only occasionally receive late updates or additions, but may be constantly read.</p>
+</li>
+<li>
+<p><strong>data on disk</strong>: The total amount of data stored on a tablet server across all disks,
+post-replication, post-compression, and post-encoding.</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_example_workload"><a class="link" href="#_example_workload">Example Workload</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The sections below perform sample calculations using the following parameters:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>200 hot replicas per tablet server</p>
+</li>
+<li>
+<p>1600 cold replicas per tablet server</p>
+</li>
+<li>
+<p>8TB of data on disk per tablet server (about 4.5GB/replica)</p>
+</li>
+<li>
+<p>512MB block cache</p>
+</li>
+<li>
+<p>40 cores per server</p>
+</li>
+<li>
+<p>limit of 32000 file descriptors per server</p>
+</li>
+<li>
+<p>a read workload with 1 frequently-scanned table with 40 columns</p>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>This workload resembles a time series use case, where the hot replicas correspond to the most recent
+range partition on time.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="memory"><a class="link" href="#memory">Memory</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The flag <code>--memory_limit_hard_bytes</code> determines the maximum amount of memory that a Kudu tablet
+server may use. The amount of memory used by a tablet server scales with data size, write workload,
+and read concurrency. The following table provides numbers that can be used to compute a rough
+estimate of memory usage.</p>
+</div>
+<table class="tableblock frame-all grid-all spread">
+<caption class="title">Table 1. Tablet Server Memory Usage</caption>
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Type</th>
+<th class="tableblock halign-left valign-top">Multiplier</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Memory required per TB of data on disk</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1.5GB per 1TB data on disk</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of memory per unit of data on disk required for
+basic operation of the tablet server.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Hot Replicas' MemRowSets and DeltaMemStores</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">minimum 128MB per hot replica</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Minimum amount of data
+to flush per MemRowSet flush. For most use cases, updates should be rare compared to inserts, so the
+DeltaMemStores should be very small.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Scans</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">256KB per column per core for read-heavy tables</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of memory used by scanners, and which
+will be constantly needed for tables which are constantly read.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Block Cache</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Fixed by <code>--block_cache_capacity_mb</code> (default 512MB)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of memory reserved for use by the
+block cache.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Using this information for the example load gives the following breakdown of memory usage:</p>
+</div>
+<table class="tableblock frame-all grid-all spread">
+<caption class="title">Table 2. Example Tablet Server Memory Usage</caption>
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Type</th>
+<th class="tableblock halign-left valign-top">Amount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">8TB data on disk</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">8TB * 1.5GB / 1TB = 12GB</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">200 hot replicas</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">200 * 128MB = 25.6GB</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1 40-column, frequently-scanned table</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">40 * 40 * 256KB = 409.6MB</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Block Cache</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>--block_cache_capacity_mb=512</code> = 512MB</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Expected memory usage</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">38.5GB</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Recommended hard limit</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">52GB</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Using this as a rough estimate of Kudu&#8217;s memory usage, select a memory limit so that the expected
+memory usage of Kudu is around 50-75% of the hard limit.</p>
+</div>
+<div class="sect2">
+<h3 id="_verifying_if_a_memory_limit_is_sufficient"><a class="link" href="#_verifying_if_a_memory_limit_is_sufficient">Verifying if a Memory Limit is sufficient</a></h3>
+<div class="paragraph">
+<p>After configuring an appropriate memory limit with <code>--memory_limit_hard_bytes</code>, run a workload and
+monitor the Kudu tablet server process&#8217;s RAM usage. The memory usage should stay around 50-75% of
+the hard limit, with occasional spikes above 75% but below 100%. If the tablet server runs above 75%
+consistently, the memory limit should be increased.</p>
+</div>
+<div class="paragraph">
+<p>Additionally, it&#8217;s also useful to monitor the logs for memory rejections, which look like:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>Service unavailable: Soft memory limit exceeded (at 96.35% of capacity)</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>and watch the memory rejections metrics:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><code>leader_memory_pressure_rejections</code></p>
+</li>
+<li>
+<p><code>follower_memory_pressure_rejections</code></p>
+</li>
+<li>
+<p><code>transaction_memory_pressure_rejections</code></p>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>Occasional rejections due to memory pressure are fine and act as backpressure to clients. Clients
+will transparently retry operations. However, no operations should time out.</p>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="file_descriptors"><a class="link" href="#file_descriptors">File Descriptors</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Processes are allotted a maximum number of open file descriptors (also referred to as fds). If a
+tablet server attempts to open too many fds, it will typically crash with a message saying something
+like "too many open files". The following table summarizes the sources of file descriptor usage in a
+Kudu tablet server process:</p>
+</div>
+<table class="tableblock frame-all grid-all spread">
+<caption class="title">Table 3. Tablet Server File Descriptor Usage</caption>
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Type</th>
+<th class="tableblock halign-left valign-top">Multiplier</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">File cache</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Fixed by <code>--block_manager_max_open_files</code> (default 40% of process maximum)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum allowed open fds reserved for use by
+the file cache.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Hot replicas</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">2 per WAL segment, 1 per WAL index</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of fds used by hot replicas. See below
+for more explanation.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Cold replicas</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">3 per cold replica</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of fds used per cold replica: 2 for the single WAL
+segment and 1 for the single WAL index.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Every replica has at least one WAL segment and at least one WAL index, and should have the same
+number of segments and indices; however, the number of segments and indices can be greater for a
+replica if one of its peer replicas is falling behind. WAL segment and index fds are closed as WALs
+are garbage collected.</p>
+</div>
+<div class="paragraph">
+<p>Using this information for the example load gives the following breakdown of file descriptor usage,
+under the assumption that some replicas are lagging and using 10 WAL segments:</p>
+</div>
+<table class="tableblock frame-all grid-all spread">
+<caption class="title">Table 4. Example Tablet Server File Descriptor Usage</caption>
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Type</th>
+<th class="tableblock halign-left valign-top">Amount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">file cache</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">40% * 32000 fds = 12800 fds</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1600 cold replicas</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1600 cold replicas * 3 fds / cold replica = 4800 fds</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">200 hot replicas</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(2 / segment * 10 segments/hot replica * 200 hot replicas) + (1 / index * 10 indices / hot replica * 200 hot replicas) = 6000 fds</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Total</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">23600 fds</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>So for this example, the tablet server process has about 32000 - 23600 = 8400 fds to spare.</p>
+</div>
+<div class="paragraph">
+<p>There is typically no downside to configuring a higher file descriptor limit if approaching the
+currently configured limit.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="threads"><a class="link" href="#threads">Threads</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Processes are allotted a maximum number of threads by the operating system, and this limit is
+typically difficult or impossible to change. Therefore, this section is more informational than
+advisory.</p>
+</div>
+<div class="paragraph">
+<p>If a Kudu tablet server&#8217;s thread count exceeds the OS limit, it will crash, usually with a message
+in the logs like "pthread_create failed: Resource temporarily unavailable". If the system thread
+count limit is exceeded, other processes on the same node may also crash.</p>
+</div>
+<div class="paragraph">
+<p>Threads and threadpools are used all over Kudu for various purposes, but the number of threads found
+in nearly all of these does not scale with load or data/tablet size; instead, the number of threads
+is either a hardcoded constant, a constant defined by a configuration parameter, or based on a
+static dimension (such as the number of CPU cores).</p>
+</div>
+<div class="paragraph">
+<p>The only exception to this is the WAL append thread, one of which exists for every "hot" replica.
+Note that all replicas may be considered hot at startup, so tablet servers' thread usage will
+generally peak when started and settle down thereafter.</p>
+</div>
+</div>
+</div>
+    </div>
+    <div class="col-md-3">
+
+  <div id="toc" data-spy="affix" data-offset-top="70">
+  <ul>
+
+      <li>
+
+          <a href="index.html">Introducing Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="release_notes.html">Kudu Release Notes</a> 
+      </li> 
+      <li>
+
+          <a href="quickstart.html">Getting Started with Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="installation.html">Installation Guide</a> 
+      </li> 
+      <li>
+
+          <a href="configuration.html">Configuring Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="kudu_impala_integration.html">Using Impala with Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="administration.html">Administering Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="troubleshooting.html">Troubleshooting Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="developing.html">Developing Applications with Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="schema_design.html">Kudu Schema Design</a> 
+      </li> 
+      <li>
+
+          <a href="security.html">Kudu Security</a> 
+      </li> 
+      <li>
+
+          <a href="transaction_semantics.html">Kudu Transaction Semantics</a> 
+      </li> 
+      <li>
+
+          <a href="background_tasks.html">Background Maintenance Tasks</a> 
+      </li> 
+      <li>
+
+          <a href="configuration_reference.html">Kudu Configuration Reference</a> 
+      </li> 
+      <li>
+
+          <a href="command_line_tools_reference.html">Kudu Command Line Tools Reference</a> 
+      </li> 
+      <li>
+
+          <a href="known_issues.html">Known Issues and Limitations</a> 
+      </li> 
+      <li>
+
+          <a href="contributing.html">Contributing to Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="export_control.html">Export Control Notice</a> 
+      </li> 
+  </ul>
+  </div>
+    </div>
+  </div>
+</div>
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/kudu/blob/d78c4caa/docs/schema_design.html
----------------------------------------------------------------------
diff --git a/docs/schema_design.html b/docs/schema_design.html
index 9039cf6..4855c37 100644
--- a/docs/schema_design.html
+++ b/docs/schema_design.html
@@ -2,7 +2,7 @@
 title: Apache Kudu Schema Design
 layout: default
 active_nav: docs
-last_updated: 'Last updated 2017-12-01 23:44:25 PST'
+last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
 ---
 <!--
 
@@ -111,6 +111,9 @@ column types include:</p>
 <p>double-precision (64-bit) IEEE-754 floating-point number</p>
 </li>
 <li>
+<p>decimal (see <a href="#decimal">Decimal Type</a> for details)</p>
+</li>
+<li>
 <p>UTF-8 encoded string (up to 64KB uncompressed)</p>
 </li>
 <li>
@@ -142,6 +145,74 @@ an explicit version or timestamp column.
 </table>
 </div>
 <div class="sect2">
+<h3 id="decimal"><a class="link" href="#decimal">Decimal Type</a></h3>
+<div class="paragraph">
+<p>The <code>decimal</code> type is a numeric data type with fixed scale and precision suitable for
+financial and other arithmetic calculations where the imprecise representation and
+rounding behavior of <code>float</code> and <code>double</code> make those types impractical. The <code>decimal</code>
+type is also useful for integers larger than int64 and cases with fractional values
+in a primary key.</p>
+</div>
+<div class="paragraph">
+<p>The <code>decimal</code> type is a parameterized type that takes precision and scale type
+attributes.</p>
+</div>
+<div class="paragraph">
+<p><strong>Precision</strong> represents the total number of digits that can be represented by the
+column, regardless of the location of the decimal point. This value must be between
+1 and 38 and has no default. For example, a precision of 4 is required to represent
+integer values up to 9999, or to represent values up to 99.99 with two fractional
+digits. You can also represent corresponding negative values, without any
+change in the precision. For example, the range -9999 to 9999 still only requires
+a precision of 4.</p>
+</div>
+<div class="paragraph">
+<p><strong>Scale</strong> represents the number of fractional digits. This value must be between 0
+and the precision. A scale of 0 produces integral values, with no fractional part.
+If precision and scale are equal, all of the digits come after the decimal point.
+For example, a decimal with precision and scale equal to 3 can represent values
+between -0.999 and 0.999.</p>
+</div>
+<div class="paragraph">
+<p><strong>Performance considerations:</strong></p>
+</div>
+<div class="paragraph">
+<p>Kudu stores each value in as few bytes as possible depending on the precision
+specified for the decimal column. For that reason it is not advised to just use
+the highest precision possible for convenience. Doing so could negatively impact
+performance, memory and storage.</p>
+</div>
+<div class="paragraph">
+<p>Before encoding and compression:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>Decimal values with precision of 9 or less are stored in 4 bytes.</p>
+</li>
+<li>
+<p>Decimal values with precision of 10 through 18 are stored in 8 bytes.</p>
+</li>
+<li>
+<p>Decimal values with precision greater than 18 are stored in 16 bytes.</p>
+</li>
+</ul>
+</div>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+The precision and scale of <code>decimal</code> columns cannot be changed by altering
+the table.
+</td>
+</tr>
+</table>
+</div>
+</div>
+<div class="sect2">
 <h3 id="encoding"><a class="link" href="#encoding">Column Encoding</a></h3>
 <div class="paragraph">
 <p>Each column in a Kudu table can be created with an encoding, based on the type
@@ -173,7 +244,7 @@ of the column.</p>
 <td class="tableblock halign-left valign-top"><p class="tableblock">bitshuffle</p></td>
 </tr>
 <tr>
-<td class="tableblock halign-left valign-top"><p class="tableblock">float, double</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">float, double, decimal</p></td>
 <td class="tableblock halign-left valign-top"><p class="tableblock">plain, bitshuffle</p></td>
 <td class="tableblock halign-left valign-top"><p class="tableblock">bitshuffle</p></td>
 </tr>
@@ -268,30 +339,40 @@ recommended to apply additional compression on top of this encoding.</p>
 <h2 id="primary-keys"><a class="link" href="#primary-keys">Primary Key Design</a></h2>
 <div class="sectionbody">
 <div class="paragraph">
-<p>Every Kudu table must declare a primary key index comprised of one or more
-columns. Primary key columns must be non-nullable, and may not be a boolean or
-floating-point type. Once set during table creation, the set of columns in the
-primary key may not be altered. Like an RDBMS primary key, the Kudu primary key
-enforces a uniqueness constraint; attempting to insert a row with the same
-primary key values as an existing row will result in a duplicate key error.</p>
+<p>Every Kudu table must declare a primary key comprised of one or more columns.
+Like an RDBMS primary key, the Kudu primary key enforces a uniqueness constraint.
+Attempting to insert a row with the same primary key values as an existing row
+will result in a duplicate key error.</p>
+</div>
+<div class="paragraph">
+<p>Primary key columns must be non-nullable, and may not be a boolean, float
+or double type.</p>
 </div>
 <div class="paragraph">
-<p>Unlike an RDBMS, Kudu does not provide an auto-incrementing column feature, so
-the application must always provide the full primary key during insert. Row
-delete and update operations must also specify the full primary key of the row
-to be changed; Kudu does not natively support range deletes or updates. The
-primary key values of a column may not be updated after the row is inserted;
-however, the row may be deleted and re-inserted with the updated value.</p>
+<p>Once set during table creation, the set of columns in the primary key may not
+be altered.</p>
+</div>
+<div class="paragraph">
+<p>Unlike an RDBMS, Kudu does not provide an auto-incrementing column feature,
+so the application must always provide the full primary key during insert.</p>
+</div>
+<div class="paragraph">
+<p>Row delete and update operations must also specify the full primary key of the
+row to be changed. Kudu does not natively support range deletes or updates.</p>
+</div>
+<div class="paragraph">
+<p>The primary key values of a column may not be updated after the row is inserted.
+However, the row may be deleted and re-inserted with the updated value.</p>
 </div>
 <div class="sect2">
 <h3 id="indexing"><a class="link" href="#indexing">Primary Key Index</a></h3>
 <div class="paragraph">
-<p>As with many traditional relational databases, Kudu&#8217;s primary key is a clustered
-index. All rows within a tablet are kept in primary key sorted order. Kudu scans
-which specify equality or range constraints on the primary key will
-automatically skip rows which can not satisfy the predicate. This allows
-individual rows to be efficiently found by specifying equality constraints on
-the primary key columns.</p>
+<p>As with many traditional relational databases, Kudu’s primary key is in a
+clustered index. All rows within a tablet are sorted by its primary key.</p>
+</div>
+<div class="paragraph">
+<p>When scanning Kudu rows, use equality or range predicates on primary key
+columns to efficiently find the rows.</p>
 </div>
 <div class="admonitionblock note">
 <table>
@@ -308,6 +389,57 @@ predicates to skip entire tablets.
 </table>
 </div>
 </div>
+<div class="sect2">
+<h3 id="Backfilling"><a class="link" href="#Backfilling">Considerations for Backfill Inserts</a></h3>
+<div class="paragraph">
+<p>This section discuss a primary key design consideration for timeseries use
+cases where the primary key is a timestamp, or the first column of the primary
+key is a timestamp.</p>
+</div>
+<div class="paragraph">
+<p>Each time a row is inserted into a Kudu table, Kudu looks up the primary key in
+the primary key index storage to check whether that primary key is already
+present in the table. If the primary key exists in the table, a "duplicate key"
+error is returned.  In the typical case where data is being inserted at
+the current time as it arrives from the data source, only a small range of
+primary keys are "hot". So, each of these "check for presence" operations is
+very fast. It hits the cached primary key storage in memory and doesn&#8217;t require
+going to disk.</p>
+</div>
+<div class="paragraph">
+<p>In the case when you load historical data, which is called "backfilling", from
+an offline data source, each row that is inserted is likely to hit a cold area
+of the primary key index which is not resident in memory and will cause one or
+more HDD disk seeks. For example, in a normal ingestion case where Kudu sustains
+a few million inserts per second, the "backfill" use case might sustain only
+a few thousand inserts per second.</p>
+</div>
+<div class="paragraph">
+<p>To alleviate the performance issue during backfilling, consider the following
+options:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>Make the primary keys more compressible.</p>
+<div class="paragraph">
+<p>For example, with the first column of a primary key being a random ID of 32-bytes,
+caching one billion primary keys would require at least 32 GB of RAM to stay in
+cache. If caching backfill primary keys from several days ago, you need to have
+several times 32 GB of memory. By changing the primary key to be more compressible,
+you increase the likelihood that the primary keys can fit in cache and thus
+reducing the amount of random disk I/Os.</p>
+</div>
+</li>
+<li>
+<p>Use SSDs for storage as random seeks are orders of magnitude faster than spinning disks.</p>
+</li>
+<li>
+<p>Change the primary key structure such that the backfill writes hit a continuous range of primary keys.</p>
+</li>
+</ul>
+</div>
+</div>
 </div>
 </div>
 <div class="sect1">
@@ -685,20 +817,6 @@ partitioning, which logically adds another dimension of partitioning.</p>
 <div class="paragraph">
 <p>Multiple alteration steps can be combined in a single transactional operation.</p>
 </div>
-<div class="admonitionblock important">
-<table>
-<tr>
-<td class="icon">
-<i class="fa icon-important" title="Important"></i>
-</td>
-<td class="content">
-<div class="title">Renaming Primary Key Columns</div>
-<a href="https://issues.apache.org/jira/browse/KUDU-1626">KUDU-1626</a>: Kudu does not yet
-support renaming primary key columns.
-</td>
-</tr>
-</table>
-</div>
 </div>
 </div>
 <div class="sect1">
@@ -810,6 +928,7 @@ altered.</p>
 <li><a href="#_the_perfect_schema">The Perfect Schema</a></li>
 <li><a href="#column-design">Column Design</a>
 <ul class="sectlevel2">
+<li><a href="#decimal">Decimal Type</a></li>
 <li><a href="#encoding">Column Encoding</a></li>
 <li><a href="#compression">Column Compression</a></li>
 </ul>
@@ -817,6 +936,7 @@ altered.</p>
 <li><a href="#primary-keys">Primary Key Design</a>
 <ul class="sectlevel2">
 <li><a href="#indexing">Primary Key Index</a></li>
+<li><a href="#Backfilling">Considerations for Backfill Inserts</a></li>
 </ul>
 </li>
 <li><a href="#partitioning">Partitioning</a>

http://git-wip-us.apache.org/repos/asf/kudu/blob/d78c4caa/docs/security.html
----------------------------------------------------------------------
diff --git a/docs/security.html b/docs/security.html
index 2ef6ee8..92c9bf1 100644
--- a/docs/security.html
+++ b/docs/security.html
@@ -2,7 +2,7 @@
 title: Apache Kudu Security
 layout: default
 active_nav: docs
-last_updated: 'Last updated 2017-12-01 23:44:25 PST'
+last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
 ---
 <!--
 
@@ -122,6 +122,22 @@ against a secured Kudu cluster where only the planner node has Kerberos
 credentials.</p>
 </div>
 </div>
+<div class="sect2">
+<h3 id="_client_authentication_to_secure_kudu_clusters"><a class="link" href="#_client_authentication_to_secure_kudu_clusters">Client Authentication to Secure Kudu Clusters</a></h3>
+<div class="paragraph">
+<p>Users running client Kudu applications must first run the <code>kinit</code> command to
+obtain a Kerberos ticket-granting ticket. For example:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="language-bash" data-lang="bash">$ kinit admin@EXAMPLE-REALM.COM</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Once authenticated, you use the same client code to read from and write to Kudu
+servers with and without Kerberos configuration.</p>
+</div>
+</div>
 </div>
 </div>
 <div class="sect1">
@@ -416,6 +432,7 @@ which require authentication or encryption.</p>
 <ul class="sectlevel2">
 <li><a href="#_internal_pki">Internal PKI</a></li>
 <li><a href="#_authentication_tokens">Authentication Tokens</a></li>
+<li><a href="#_client_authentication_to_secure_kudu_clusters">Client Authentication to Secure Kudu Clusters</a></li>
 </ul>
 </li>
 <li><a href="#_scalability">Scalability</a></li>

http://git-wip-us.apache.org/repos/asf/kudu/blob/d78c4caa/docs/transaction_semantics.html
----------------------------------------------------------------------
diff --git a/docs/transaction_semantics.html b/docs/transaction_semantics.html
index 1687924..44ba1fa 100644
--- a/docs/transaction_semantics.html
+++ b/docs/transaction_semantics.html
@@ -2,7 +2,7 @@
 title: Transaction Semantics in Apache Kudu
 layout: default
 active_nav: docs
-last_updated: 'Last updated 2017-03-08 17:29:48 PST'
+last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
 ---
 <!--
 
@@ -65,9 +65,9 @@ by the system, after completion.</p>
 </li>
 <li>
 <p><strong>Scans</strong> are read operations that can traverse multiple tablets and read information
-with some consistency or correctness guarantees. Scans can perform time-travel reads, i.e.
-the user is able to set a scan timestamp in the past and get back results that reflect
-the state of the storage engine at that point in time.</p>
+with different levels of consistency or correctness guarantees. Scans can perform
+time-travel reads, i.e. the user is able to set a scan timestamp in the past and
+get back results that reflect the state of the storage engine at that point in time.</p>
 </li>
 </ul>
 </div>
@@ -164,7 +164,7 @@ different tablets can be observed by clients.</p>
 </div>
 <div class="paragraph">
 <p>Kudu, like Spanner, was designed to be externally consistent <a href="#5">[5]</a>, preserving consistency
-even when operations span multiple tablets and even multiple data centers. In practice this
+when operations span multiple tablets and even multiple data centers. In practice this
 means that, if a write operation changes item <em>x</em> at tablet <em>A</em>, and a following write
 operation changes item <em>y</em> at tablet <em>B</em>, you might want to enforce that if
 the change to <em>y</em> is observed, the change to <em>x</em> must also be observed. There
@@ -259,7 +259,7 @@ the user. The mode may be selected as follows:</p>
 <dl>
 <dt class="hdlist1">Java Client</dt>
 <dd>
-<p>Call <code>KuduScannerBuilder#setReadMode(&#8230;&#8203;)</code></p>
+<p>Call <code>KuduScannerBuilder#readMode(&#8230;&#8203;)</code></p>
 </dd>
 <dt class="hdlist1">C++ Client</dt>
 <dd>
@@ -301,7 +301,39 @@ up-to-date view of the database. In that case, you might choose <code>READ_AT_SN
 and select a timestamp that is a few seconds in the past when the scan starts. On
 the other hand, a machine learning workload that is not ingesting the whole data
 set and is already statistical in nature might not require the scan to be repeatable,
-so you might choose <code>READ_LATEST</code> instead.</p>
+so you might choose <code>READ_LATEST</code> instead for better scan performance.</p>
+</div>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>Kudu also provides replica selection API for users to choose at which replica the
+scan should be performed:</p>
+</div>
+<div class="dlist">
+<dl>
+<dt class="hdlist1">Java Client</dt>
+<dd>
+<p>Call <code>KuduScannerBuilder#replicaSelection(&#8230;&#8203;)</code></p>
+</dd>
+<dt class="hdlist1">C++ Client</dt>
+<dd>
+<p>Call <code>KuduScanner::SetSelection(&#8230;&#8203;)</code></p>
+</dd>
+</dl>
+</div>
+<div class="paragraph">
+<p>This API is a means to control locality and, in some cases, latency. The replica
+selection API has no effect on the consistency guarantees, which will hold no matter
+which replica is selected.</p>
+</div>
+</td>
+</tr>
+</table>
 </div>
 </div>
 </div>
@@ -313,7 +345,7 @@ so you might choose <code>READ_LATEST</code> instead.</p>
 in some situations, at the moment. Below are the details and next, some recommendations.</p>
 </div>
 <div class="sect2">
-<h3 id="known_issues_scans"><a class="link" href="#known_issues_scans">Reads (Scans)</a></h3>
+<h3 id="known_issues_scans"><a class="link" href="#known_issues_scans">Writes</a></h3>
 <div class="ulist">
 <ul>
 <li>
@@ -325,7 +357,7 @@ is discouraged in production environments.</p>
 </div>
 </div>
 <div class="sect2">
-<h3 id="_writes"><a class="link" href="#_writes">Writes</a></h3>
+<h3 id="_reads_scans"><a class="link" href="#_reads_scans">Reads (Scans)</a></h3>
 <div class="ulist">
 <ul>
 <li>
@@ -358,7 +390,7 @@ See <a href="https://issues.apache.org/jira/browse/KUDU-1767">KUDU-1767</a>.</p>
 <li>
 <p>If repeatable snapshot reads are a requirement, use <code>READ_AT_SNAPSHOT</code>
 with a timestamp that is slightly in the past (between 2-5 seconds, ideally).
-This will circumvent the anomaly described in <a href="#known_issues_scans">Reads (Scans)</a>. Even when the
+This will circumvent the anomaly described in <a href="#known_issues_scans">Writes</a>. Even when the
 anomaly has been addressed, back-dating the timestamp will always make scans
 faster, since they are unlikely to block.</p>
 </li>
@@ -496,8 +528,8 @@ frequently.
 <li><a href="#_read_operations_scans">Read Operations (Scans)</a></li>
 <li><a href="#known_issues">Known Issues and Limitations</a>
 <ul class="sectlevel2">
-<li><a href="#known_issues_scans">Reads (Scans)</a></li>
-<li><a href="#_writes">Writes</a></li>
+<li><a href="#known_issues_scans">Writes</a></li>
+<li><a href="#_reads_scans">Reads (Scans)</a></li>
 <li><a href="#recommendations">Recommendations</a></li>
 </ul>
 </li>

http://git-wip-us.apache.org/repos/asf/kudu/blob/d78c4caa/docs/troubleshooting.html
----------------------------------------------------------------------
diff --git a/docs/troubleshooting.html b/docs/troubleshooting.html
index cdd006d..d259ed6 100644
--- a/docs/troubleshooting.html
+++ b/docs/troubleshooting.html
@@ -2,7 +2,7 @@
 title: Apache Kudu Troubleshooting
 layout: default
 active_nav: docs
-last_updated: 'Last updated 2017-12-01 23:44:25 PST'
+last_updated: 'Last updated 2018-03-21 12:01:40 CDT'
 ---
 <!--
 
@@ -121,19 +121,8 @@ the user will see a message such as:</p>
 <pre>Sep 17, 8:32:31.135 PM FATAL tablet_server_main.cc:38 Check failed: _s.ok() Bad status: Service unavailable: Cannot initialize clock: Cannot initialize HybridClock. Clock synchronized but error was too high (11711000 us).</pre>
 </div>
 </div>
-<div class="admonitionblock tip">
-<table>
-<tr>
-<td class="icon">
-<i class="fa icon-tip" title="Tip"></i>
-</td>
-<td class="content">
-If NTP is installed the user can monitor the synchronization status by running
-<code>ntptime</code>. The relevant value is what is reported for <code>maximum error</code>.
-</td>
-</tr>
-</table>
-</div>
+<div class="sect3">
+<h4 id="_installing_ntp"><a class="link" href="#_installing_ntp">Installing NTP</a></h4>
 <div class="paragraph">
 <p>To install NTP, use the appropriate command for your operating system:</p>
 </div>
@@ -184,6 +173,93 @@ If NTP is installed the user can monitor the synchronization status by running
 </tr>
 </tbody>
 </table>
+</div>
+<div class="sect3">
+<h4 id="_monitoring_ntp_status"><a class="link" href="#_monitoring_ntp_status">Monitoring NTP Status</a></h4>
+<div class="paragraph">
+<p>When NTP is installed, you can monitor the synchronization status by running
+<code>ntptime</code>. For example, a healthy system may report:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>ntp_gettime() returns code 0 (OK)
+  time de24c0cf.8d5da274  Tue, Feb  6 2018 16:03:27.552, (.552210980),
+  maximum error 224455 us, estimated error 383 us, TAI offset 0
+ntp_adjtime() returns code 0 (OK)
+  modes 0x0 (),
+  offset 1279.543 us, frequency 2.500 ppm, interval 1 s,
+  maximum error 224455 us, estimated error 383 us,
+  status 0x2001 (PLL,NANO),
+  time constant 10, precision 0.001 us, tolerance 500 ppm,</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>In particular, note the following most important pieces of output:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><code>maximum error 22455 us</code>: this value is well under the 10-second maximum error required
+by Kudu.</p>
+</li>
+<li>
+<p><code>status 0x2001 (PLL,NANO)</code>: this indicates a healthy synchronization status.</p>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>In contrast, a system without NTP properly configured and running will output
+something like the following:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>ntp_gettime() returns code 5 (ERROR)
+  time de24c240.0c006000  Tue, Feb  6 2018 16:09:36.046, (.046881),
+  maximum error 16000000 us, estimated error 16000000 us, TAI offset 0
+ntp_adjtime() returns code 5 (ERROR)
+  modes 0x0 (),
+  offset 0.000 us, frequency 2.500 ppm, interval 1 s,
+  maximum error 16000000 us, estimated error 16000000 us,
+  status 0x40 (UNSYNC),
+  time constant 10, precision 1.000 us, tolerance 500 ppm,</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Note the <code>UNSYNC</code> status and the 16-second maximum error.</p>
+</div>
+<div class="paragraph">
+<p>If more detailed information is needed, the <code>ntpq</code> or <code>ntpdc</code> tools
+can be used to dump further information about which network time servers
+are currently acting as sources:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>$ ntpq -n -c opeers
+     remote           local      st t when poll reach   delay   offset    disp
+==============================================================================
+ 0.0.0.0         0.0.0.0         16 p    -   64    0    0.000    0.000 16000.0
+ 0.0.0.0         0.0.0.0         16 p    -   64    0    0.000    0.000 16000.0
+ 0.0.0.0         0.0.0.0         16 p    -   64    0    0.000    0.000 16000.0
+ 0.0.0.0         0.0.0.0         16 p    -   64    0    0.000    0.000 16000.0
+ 0.0.0.0         0.0.0.0         16 p    -   64    0    0.000    0.000 16000.0
+-108.59.2.24     10.16.2.89       2 u    3   64    3   74.380    0.321  62.992
+-208.82.104.205  10.16.2.89       2 u    5   64    3   52.654   -4.054  62.965
+#192.96.202.120  10.16.2.89       2 u    1   64    3   74.737    6.538  62.988
+#69.10.161.7     10.16.2.89       3 u    5   64    3   28.353   -1.967  62.960
+-173.255.206.154 10.16.2.89       3 u    -   64    3   42.906   -3.127  62.996
+-69.195.159.158  10.16.2.89       2 u    1   64    3   52.543   -4.788  62.987
+*216.218.254.202 10.16.2.89       1 u    5   64    3    2.567    0.053  62.974
+-129.250.35.250  10.16.2.89       2 u    3   64    3    2.603    0.256  62.985
++45.76.244.193   10.16.2.89       2 u    5   64    3   19.522    0.188  62.969
+-69.89.207.199   10.16.2.89       2 u    5   64    3   66.687   -0.395  62.967
+-171.66.97.126   10.16.2.89       1 u    1   64    3   12.627   -3.572  62.963
+#66.228.42.59    10.16.2.89       4 u    1   64    3   72.143    4.034  62.971
+ 91.189.89.198   10.16.2.89       2 u    5   64    3  135.329    3.069 3937.74
+#162.210.111.4   10.16.2.89       2 u    -   64    3   29.572    6.849  62.966
++199.102.46.80   10.16.2.89       1 u    3   64    3   57.022    0.111  63.386
+ 91.189.89.199   10.16.2.89       2 u    4   64    3  138.269    3.228 3937.98</pre>
+</div>
+</div>
 <div class="admonitionblock tip">
 <table>
 <tr>
@@ -191,18 +267,121 @@ If NTP is installed the user can monitor the synchronization status by running
 <i class="fa icon-tip" title="Tip"></i>
 </td>
 <td class="content">
-NTP requires a network connection and may take a few minutes to synchronize the clock.
-In some cases a spotty network connection may make NTP report the clock as unsynchronized.
-A common, though temporary, workaround for this is to restart NTP with one of the commands above.
+Depending on the specific version of NTP, the correct command may be either
+<code>ntpq -n -c opeers</code> or <code>ntpq -n -c lpeers</code>.
 </td>
 </tr>
 </table>
 </div>
+<div class="sidebarblock">
+<div class="content">
+<div class="paragraph">
+<div class="title">Using <code>chrony</code> for time synchronization</div>
+<p>Some operating systems offer <code>chrony</code> as an alternative to <code>ntpd</code> for network time
+synchronization. Kudu has been tested most thoroughly using <code>ntpd</code> and use of
+<code>chrony</code> is considered experimental.</p>
+</div>
 <div class="paragraph">
-<p>If the clock is being reported as synchronized by NTP, but the maximum error is too high,
-the user can increase the threshold to a higher value by setting the above
-mentioned flag. For example to increase the possible maximum error to
-20 seconds the flag should be set like: <code>--max_clock_sync_error_usec=20000000</code></p>
+<p>In order to use <code>chrony</code> for synchronization, <code>chrony.conf</code> must be configured
+with the <code>rtcsync</code> option.</p>
+</div>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_ntp_configuration_best_practices"><a class="link" href="#_ntp_configuration_best_practices">NTP Configuration Best Practices</a></h4>
+<div class="paragraph">
+<p>In order to provide stable time synchronization with low maximum error, follow
+these best NTP configuration best practices.</p>
+</div>
+<div class="paragraph">
+<p><strong>Always configure at least four time sources for NTP.</strong> In addition to providing
+redundancy in case one or more time sources becomes unavailable, The NTP protocol is
+designed to increase its accuracy with a diversity of sources. Even if your organization
+provides one or more local time servers, configuring additional remote servers is highly
+recommended for a robust setup.</p>
+</div>
+<div class="paragraph">
+<p><strong>Pick servers in your server&#8217;s local geography.</strong> For example, if your servers are located
+in Europe, pick servers from the European NTP pool. If your servers are running in a public
+cloud environment, consult the cloud provider&#8217;s documentation for a recommended NTP setup.
+Many cloud providers offer highly accurate clock synchronization as a service.</p>
+</div>
+<div class="paragraph">
+<p><strong>Use the <code>iburst</code> option for faster synchronization at startup</strong>. The <code>iburst</code> option
+instructs <code>ntpd</code> to send an initial "burst" of time queries at startup. This typically
+results in a faster time synchronization when a machine restarts.</p>
+</div>
+<div class="paragraph">
+<p>An example NTP server list may appear as follows:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre># Use my organization's internal NTP servers.
+server ntp1.myorg.internal iburst
+server ntp2.myorg.internal iburst
+# Provide several public pool servers from the US pool for
+# redundancy and robustness.
+server 0.pool.us.ntp.org iburst
+server 1.pool.us.ntp.org iburst
+server 2.pool.us.ntp.org iburst
+server 3.pool.us.ntp.org iburst</pre>
+</div>
+</div>
+<div class="admonitionblock tip">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-tip" title="Tip"></i>
+</td>
+<td class="content">
+After configuring NTP, use the <code>ntpq</code> tool described above to verify that <code>ntpd</code> was
+able to connect to a variety of peers. If no public peers appear, it is possiblbe that
+the NTP protocol is being blocked by a firewall or other network connectivity issue.
+</td>
+</tr>
+</table>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_troubleshooting_ntp_stability_problems"><a class="link" href="#_troubleshooting_ntp_stability_problems">Troubleshooting NTP Stability Problems</a></h4>
+<div class="paragraph">
+<p>As of Kudu 1.6.0, Kudu daemons are able to continue to operate during a brief loss of
+NTP synchronization. If NTP synchronization is lost for several hours, however, daemons
+may crash. If a daemon crashes due to NTP synchronization issues, consult the <code>ERROR</code> log
+for a dump of related information which may help to diagnose the issue.</p>
+</div>
+<div class="admonitionblock tip">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-tip" title="Tip"></i>
+</td>
+<td class="content">
+Kudu 1.5.0 and earlier versions were less resilient to brief NTP outages. In
+addition, they contained a <a href="https://issues.apache.org/jira/browse/KUDU-2209">bug</a>
+which could cause Kudu to incorrectly measure the maximum error, resulting in
+crashes. If you experience crashes related to clock synchronization on these
+earlier versions of Kudu and it appears that the system&#8217;s NTP configuration is correct,
+consider upgrading to Kudu 1.6.0 or later.
+</td>
+</tr>
+</table>
+</div>
+<div class="admonitionblock tip">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-tip" title="Tip"></i>
+</td>
+<td class="content">
+NTP requires a network connection and may take a few minutes to synchronize the clock
+at startup. In some cases a spotty network connection may make NTP report the clock as unsynchronized.
+A common, though temporary, workaround for this is to restart NTP with one of the commands above.
+</td>
+</tr>
+</table>
+</div>
 </div>
 </div>
 </div>
@@ -474,6 +653,65 @@ Generally, the recommended ratio of maintenance manager threads to data director
 </div>
 </div>
 <div class="sect2">
+<h3 id="heap_sampling"><a class="link" href="#heap_sampling">Heap Sampling</a></h3>
+<div class="paragraph">
+<p>For advanced debugging of memory usage, administrators may enable heap sampling on Kudu daemons.
+This allows Kudu developers to associate memory usage with the specific lines of code and data
+structures responsible. When reporting a bug related to memory usage or an apparent memory leak,
+heap profiling can give quantitative data to pinpoint the issue.</p>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+Heap sampling is an advanced troubleshooting technique and may cause performance
+degradation or instability of the Kudu service. Currently it is not recommended to enable this
+in a production environment unless specifically requested by the Kudu development team.
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>To enable heap sampling on a Kudu daemon, pass the flag <code>--heap-sample-every-n-bytes=524588</code>.
+If heap sampling is enabled, the current sampled heap occupancy can be retrieved over HTTP
+by visiting <code><a href="http://tablet-server.example.com:8050/pprof/heap" class="bare">http://tablet-server.example.com:8050/pprof/heap</a></code> or
+<code><a href="http://master.example.com:8051/pprof/heap" class="bare">http://master.example.com:8051/pprof/heap</a></code>. The output is a machine-readable dump of the
+stack traces with their associated heap usage.</p>
+</div>
+<div class="paragraph">
+<p>Rather than visiting the heap profile page directly in a web browser, it is typically
+more useful to use the <code>pprof</code> tool that is distributed as part of the <code>gperftools</code>
+open source project. For example, a developer with a local build tree can use the
+following command to collect the sampled heap usage and output an SVG diagram:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>thirdparty/installed/uninstrumented/bin/pprof -svg  'http://localhost:8051/pprof/heap' &gt; /tmp/heap.svg</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The resulting SVG may be visualized in a web browser or sent to the Kudu community to help
+troubleshoot memory occupancy issues.</p>
+</div>
+<div class="admonitionblock tip">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-tip" title="Tip"></i>
+</td>
+<td class="content">
+Heap samples contain only summary information about allocations and do not contain any
+<em>data</em> from the heap. It is safe to share heap samples in public without fear of exposing
+confidential or sensitive data.
+</td>
+</tr>
+</table>
+</div>
+</div>
+<div class="sect2">
 <h3 id="disk_issues"><a class="link" href="#disk_issues">Disk Issues</a></h3>
 <div class="paragraph">
 <p>When Kudu starts, it checks each configured data directory, expecting either for all to be
@@ -481,7 +719,7 @@ initialized or for all to be empty. If a server fails to start with a log messag
 </div>
 <div class="listingblock">
 <div class="content">
-<pre>Check failed: _s.ok() Bad status: Already present: Could not create new FS layout: FSManager root is not empty: /data0/kudu/data</pre>
+<pre>Check failed: _s.ok() Bad status: Already present: FS layout already exists; not overwriting existing layout: FSManager roots already exist: /data0/kudu/data</pre>
 </div>
 </div>
 <div class="paragraph">
@@ -558,6 +796,7 @@ There is no workaround for Hive users. Spark users need to create temporary tabl
 <li><a href="#_rpc_timeout_traces">RPC Timeout Traces</a></li>
 <li><a href="#_kernel_stack_watchdog_traces">Kernel Stack Watchdog Traces</a></li>
 <li><a href="#memory_limits">Memory Limits</a></li>
+<li><a href="#heap_sampling">Heap Sampling</a></li>
 <li><a href="#disk_issues">Disk Issues</a></li>
 </ul>
 </li>

http://git-wip-us.apache.org/repos/asf/kudu/blob/d78c4caa/releases/1.6.0/index.md
----------------------------------------------------------------------
diff --git a/releases/1.6.0/index.md b/releases/1.6.0/index.md
index 169a080..6a9e542 100644
--- a/releases/1.6.0/index.md
+++ b/releases/1.6.0/index.md
@@ -33,8 +33,8 @@ See the [Kudu 1.6.0 Release Notes](docs/release_notes.html).
 Downloads of Kudu 1.6.0 are available in the following formats:
 
 * [Kudu 1.6.0 source tarball](https://archive.apache.org/dist/kudu/1.6.0/apache-kudu-1.6.0.tar.gz)
-  ([SHA1](https://www.apache.org/dist/kudu/1.6.0/apache-kudu-1.6.0.tar.gz.sha),
-  [Signature](https://www.apache.org/dist/kudu/1.6.0/apache-kudu-1.6.0.tar.gz.asc))
+  ([SHA1](https://archive.apache.org/dist/kudu/1.6.0/apache-kudu-1.6.0.tar.gz.sha),
+  [Signature](https://archive.apache.org/dist/kudu/1.6.0/apache-kudu-1.6.0.tar.gz.asc))
 
 You can use the [KEYS file](https://www.apache.org/dist/kudu/KEYS) to verify the included GPG signature.
 


Mime
View raw message