kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jdcry...@apache.org
Subject [01/45] incubator-kudu git commit: Add 0.8.0 docs
Date Mon, 11 Apr 2016 16:28:06 GMT
Repository: incubator-kudu
Updated Branches:
  refs/heads/gh-pages 073938d7a -> c78a2f80e


http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/c78a2f80/releases/0.8.0/docs/transaction_semantics.html
----------------------------------------------------------------------
diff --git a/releases/0.8.0/docs/transaction_semantics.html b/releases/0.8.0/docs/transaction_semantics.html
new file mode 100644
index 0000000..feb7835
--- /dev/null
+++ b/releases/0.8.0/docs/transaction_semantics.html
@@ -0,0 +1,488 @@
+---
+title: Transaction Semantics in Apache Kudu (incubating)
+layout: default
+active_nav: docs
+last_updated: 'Last updated 2016-04-11 08:31:49 PDT'
+---
+<!--
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+
+<div class="container">
+  <div class="row">
+    <div class="col-md-9">
+
+<h1>Transaction Semantics in Apache Kudu (incubating)</h1>
+      <div id="preamble">
+<div class="sectionbody">
+<div class="sidebarblock">
+<div class="content">
+<div class="paragraph">
+<p>This is a brief introduction to Kudu&#8217;s transaction and consistency semantics. For an
+in-depth technical exposition of most of what is mentioned here, and why it is correct,
+see the technical report <a href="#1">[1]</a>.</p>
+</div>
+</div>
+</div>
+<div class="paragraph">
+<p>Kudu&#8217;s transactional semantics and architecture are inspired by state-of-the-art
+systems such as Spanner <a href="#2">[2]</a> and Calvin <a href="#3">[3]</a>. Kudu builds upon decades of database
+research. The core philosophy is to make the lives of developers easier by providing transactions with
+simple, strong semantics, without sacrificing performance or the ability to tune to different
+requirements.</p>
+</div>
+<div class="paragraph">
+<p>Kudu is designed to eventually be fully ACID, however, multi-tablet transactions are not
+yet implemented. As such, this discussion focuses on single-tablet write operations, and only
+briefly touches multi-tablet reads. Eventually Kudu will support fully strict-serializable
+semantics. In fact it already does in a limited context, but not all corner cases are covered
+as this is still a work in progress.</p>
+</div>
+<div class="paragraph">
+<p>Kudu currently allows the following operations:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><strong>Write operations</strong> are sets of rows to be inserted, updated, or deleted in the storage
+engine, in a single tablet with multiple replicas. Write operations do not have separate
+"read sets" i.e. they do not scan existing data before performing the write. Each write
+is only concerned with previous state of the rows that are about to change.
+Writes are not  "committed" explicitly by the user. Instead, they are committed automatically
+by the system, after completion.</p>
+</li>
+<li>
+<p><strong>Scans</strong> are read operations that can traverse multiple tablets and read information
+with some consistency or correctness guarantees. Scans can perform time-travel reads, i.e.
+the user is able to set a scan timestamp in the past and get back results that reflect
+the state of the storage engine at that point in time.</p>
+</li>
+</ul>
+</div>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="title">Before We Begin</div>
+<div class="ulist">
+<ul>
+<li>
+<p>The term <em>timestamp</em> is mentioned several times to illustrate the
+functionality, but <em>timestamp</em> is an internal concept mostly invisible to users,
+except when setting timestamp on a <code>KuduScanner</code>.</p>
+</li>
+<li>
+<p>We generally refer to methods and classes of the <em>async java</em> client. While the C++
+client mostly has analogous methods and classes, parity between the APIs is still
+a work in progress. At times, we may refer specifically to the C++ client.</p>
+</li>
+</ul>
+</div>
+</td>
+</tr>
+</table>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_single_tablet_write_operations"><a class="link" href="#_single_tablet_write_operations">Single tablet write operations</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Kudu employs <em>Multiversion Concurrency Control (MVCC)</em> and the <em>Raft consensus</em> algorithm <a href="#4">[4]</a>.
+Each write operation in Kudu must go through the tablet&#8217;s leader.</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>The leader acquires all locks for the rows that it will change.</p>
+</li>
+<li>
+<p>The leader assigns the write a timestamp before the write is submitted for
+replication. This timestamp will be the write&#8217;s "tag" in MVCC.</p>
+</li>
+<li>
+<p>After a majority of replicas acknowledges the change, the actual rows are changed.</p>
+</li>
+<li>
+<p>After the changes are complete, they are made visible to concurrent writes
+and reads, atomically.</p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>All replicas of a tablet observe the same order of operations and if a write
+operation is assigned timestamp <em>n</em> and changes row <em>x</em>, a second write operation
+at timestamp <em>m &gt; n</em> is guaranteed to see the new value of <em>x</em>.</p>
+</div>
+<div class="paragraph">
+<p>This strict ordering of lock acquisition and timestamp assignment is enforced to be
+consistent across all replicas of a tablet through consensus. Therefore, write operations
+are totally ordered with regard to clock-assigned timestamps, relative to other writes
+in the same tablet. In other words, writes have strict-serializable semantics,
+though in an admittedly limited context. See this
+<a href="http://www.bailis.org/blog/linearizability-versus-serializability">blog post</a>
+for a little more context regarding what these semantics mean.</p>
+</div>
+<div class="paragraph">
+<p>While Isolated and Durable in an ACID sense, write operations are not yet fully Atomic.
+The failure of a single write in a batch operation does not roll back the operation,
+but produces per-row errors.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_writing_to_multiple_tablets"><a class="link" href="#_writing_to_multiple_tablets">Writing to multiple tablets</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Kudu does not yet support transactions that span multiple tablets. However,
+consistent snapshot reads are possible (with caveats in the current implementation)
+as explained below.</p>
+</div>
+<div class="paragraph">
+<p>Writes to a Kudu client are optionally buffered in memory until they are flushed and sent
+to the server. During the client&#8217;s session flush, the rows for each tablet are batched
+together, and sent to the tablet server which hosts the leader replica of the tablet.
+Since there are no inter-tablet transactions, each of these batches represents a single,
+independent write operation with its own timestamp.
+However you have the option to impose some constraints on the assigned timestamps
+and on how writes to different tablets can be observed by clients.</p>
+</div>
+<div class="paragraph">
+<p>Kudu, like Spanner, was designed to be externally consistent <a href="#5">[5]</a>, preserving consistency
+even when operations span multiple tablets and even multiple data centers. In practice this
+means that, if a write operation changes item <em>x</em> at tablet <em>A</em>, and a following write
+operation changes item <em>y</em> at tablet <em>B</em>, you might want to enforce that if
+the change to <em>y</em> is observed, the change to <em>x</em> must also be observed. There
+are many examples where this can be important. For example,  if Kudu is
+storing clickstreams for further analysis, and two clicks follow each other but
+are stored in different tablets, subsequent clicks should be assigned subsequent
+timestamps so that the causal relationship between them is captured.</p>
+</div>
+<div class="paragraph">
+<div class="title"><code>CLIENT_PROPAGATED</code> Consistency</div>
+<p>Kudu&#8217;s default external consistency mode is called <code>CLIENT_PROPAGATED</code>.
+See <a href="#1">[1]</a> for an extensive explanation on how it works. In brief, this mode causes writes
+from <em>a single client</em> to be automatically externally consistent. In this mode, writes are only externally
+consistent from the perspective of a single client. In the clickstream scenario above,
+if the two clicks are submitted by different client instances, the application must
+manually propagate timestamps from one client to the other for the causal relationship
+to be captured.</p>
+</div>
+<div class="paragraph">
+<p><code>CLIENT_PROPAGATED</code> consistency is currently only available on the java client
+and is exposed through the <code>AsyncKuduClient#getLastPropagatedTimestamp()</code> and
+<code>AsyncKuduClient#setLastPropagatedTimestamp()</code> methods.</p>
+</div>
+<div class="paragraph">
+<div class="title"><code>Commit Wait</code> Consistency</div>
+<p>Kudu also implements an experimental implementation of an external consistency
+model used in Google&#8217;s Spanner , called <code>Commit Wait</code>. <code>Commit Wait</code> works
+by tightly synchronizing the clocks on all machines in the cluster. Then, when a
+write occurs, timestamps are assigned and the results of the write are not made
+visible until enough time has passed so that no other machine in the cluster could
+possibly assign a lower timestamp to a following write.</p>
+</div>
+<div class="paragraph">
+<p>For the moment, Kudu&#8217;s experimental implementation of <code>Commit Wait</code> is only available
+in the java client, by setting <code>KuduSession#setExternalConsistencyMode()</code>
+to <code>COMMIT_WAIT</code>. When using this mode, the latency of writes is tightly
+tied to the accuracy of clocks on all the cluster hosts, and using this mode
+with loose clock synchronization causes writes to take a long time to complete or even time
+out. See <a href="#known_issues">Known Issues and Limitations</a>.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_read_operations_scans"><a class="link" href="#_read_operations_scans">Read Operations (Scans)</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Scans are read operations performed by clients that may span one or more rows across
+one or more tablets. When a server receives a scan, it takes a snapshot of the MVCC
+state and then proceeds in one of two ways depending on the read mode selected by
+the user by means of the <code>KuduScanner::SetReadMode()</code> method.</p>
+</div>
+<div class="dlist">
+<dl>
+<dt class="hdlist1"><code>READ_LATEST</code></dt>
+<dd>
+<p>This is the default read mode. The server takes a snapshot of
+the MVCC state and proceeds with the read immediately. Reads in this mode only yield
+'Read Committed' isolation.</p>
+</dd>
+<dt class="hdlist1"><code>READ_AT_SNAPSHOT</code></dt>
+<dd>
+<p>In this read mode, scans are consistent and repeatable. A
+timestamp for the snapshot is selected either by the server, or set
+explicitly by the user through <code>KuduScanner::SetSnapshotMicros()</code>. Explicitly setting
+the timestamp is recommended; see <a href="#recommendations">Recommendations</a>. The server waits until this
+timestamp is 'safe' (until all write operations that have a lower timestamp have
+completed and are visible). This delay, coupled with an external consistency method,
+will eventually allow Kudu to have full <code>strict-serializable</code> semantics for reads
+and writes. This is still a work in progress and some anomalies are still possible
+(see <a href="#known_issues">Known Issues and Limitations</a>). Only scans in this mode can be fault-tolerant.</p>
+</dd>
+</dl>
+</div>
+<div class="paragraph">
+<p>Selecting between read modes requires balancing the trade-offs and making a choice
+that fits your workload. For instance, a reporting application that needs to
+scan the entire database might need to perform careful accounting operations, so that
+scan may need to be fault-tolerant, but probably doesn&#8217;t require a to-the-microsecond
+up-to-date view of the database. In that case, you might choose 'READ_AT_SNAPSHOT'
+and select a timestamp that is a few seconds in the past when the scan starts. On
+the other hand, a machine learning workload that is not ingesting the whole data
+set and is already statistical in nature might not require the scan to be repeatable,
+so you might choose <code>READ_LATEST</code> instead.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="known_issues"><a class="link" href="#known_issues">Known Issues and Limitations</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>We plan to fix the following issues. Monitor the linked JIRAs for progress.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_serialization"><a class="link" href="#_serialization">Serialization</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>There are several gaps and corner cases that prevent Kudu from being fully strictly-serializable
+in some situations, at the moment. Below are the details and next, some recommendations.</p>
+</div>
+<div class="sect2">
+<h3 id="known_issues_scans"><a class="link" href="#known_issues_scans">Scans</a></h3>
+<div class="ulist">
+<ul>
+<li>
+<p>Support for <code>COMMIT_WAIT</code> is experimental and requires careful tuning of the
+time-synchronization protocol, such as NTP (Network Time Protocol).</p>
+</li>
+<li>
+<p>Support for externally-consistent write modes is only fully available in the Java
+API at this time. (see <a href="https://issues.cloudera.org/browse/KUDU-1187">KUDU-1187</a>)</p>
+</li>
+<li>
+<p>In some rare circumstances, the <code>READ_AT_SNAPSHOT</code> scan mode may yield anomalous,
+non-repeatable reads.</p>
+<div class="ulist">
+<ul>
+<li>
+<p>When scanning a replica at a snapshot, the replica may not have received all the writes
+from the leader and might reply immediately, yielding a non-repeatable read (see <a href="https://issues.cloudera.org/browse/KUDU-798">KUDU-798</a>).</p>
+</li>
+<li>
+<p>On a leader change, scans at a snapshot whose timestamp is beyond the last
+write may also yield non-repeatable reads (see <a href="https://issues.cloudera.org/browse/KUDU-1188">KUDU-1188</a>). See <a href="#recommendations">Recommendations</a> for a workaround.</p>
+</li>
+<li>
+<p>When performing multi-tablet scans without selecting a snapshot timestamp (see <a href="https://issues.cloudera.org/browse/KUDU-1189">KUDU-1189</a>).</p>
+</li>
+</ul>
+</div>
+</li>
+<li>
+<p>Impala scans are currently performed as <code>READ_LATEST</code> and have no consistency
+guarantees.</p>
+</li>
+</ul>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_writes"><a class="link" href="#_writes">Writes</a></h3>
+<div class="ulist">
+<ul>
+<li>
+<p>When a write fails with a timeout or is aborted, it is possible that it may
+actually be committed. Kudu is currently missing a way to determine if a particular
+timed-out write ever actually succeeded. On a retry, the write may succeed but
+may also generate errors if some rows have already been inserted, or deleted (see <a href="https://issues.cloudera.org/browse/KUDU-568">KUDU-568</a>).</p>
+</li>
+<li>
+<p>When a delete is performed to a row that has already been flushed, and the row is reinserted
+all history is reset (see <a href="https://issues.cloudera.org/browse/KUDU-237">KUDU-237</a>).
+This is not the case for rows that haven&#8217;t been flushed yet and still reside in memory.</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="recommendations"><a class="link" href="#recommendations">Recommendations</a></h2>
+<div class="sectionbody">
+<div class="ulist">
+<ul>
+<li>
+<p>If repeatable snapshot reads are a requirement, use <code>READ_AT_SNAPSHOT</code>
+with a timestamp that is slightly in the past (between 2-5 seconds, ideally).
+This will circumvent the anomalies described in <a href="#known_issues_scans">Scans</a>. Even when the
+anomalies have been addressed, back-dating the timestamp will always make scans
+faster, since they are unlikely to block.</p>
+</li>
+<li>
+<p>If external consistency is a requirement and you decide to use <code>Commit Wait</code>, the
+time-synchronization protocol needs to be tuned carefully. Each transaction will wait
+2x the maximum clock error at the time of execution, which is usually in the 100 msec.
+to 1 sec. range with the default settings, maybe more. Thus, transactions would take at least
+200 msec. to 2 sec. to complete when using the default settings and may even time out.</p>
+<div class="ulist">
+<ul>
+<li>
+<p>A local server should be used as a time server. We&#8217;ve performed experiments using the default
+NTP time source available in a Google Compute Engine data center and were able to obtain
+a reasonable tight max error bound, usually varying between 12-17 milliseconds.</p>
+</li>
+<li>
+<p>The following parameters should be adjusted in <code>/etc/ntp.conf</code> to tighten the maximum error:</p>
+<div class="ulist">
+<ul>
+<li>
+<p><code>server my_server.org iburst minpoll 1 maxpoll 8</code></p>
+</li>
+<li>
+<p><code>tinker dispersion 500</code></p>
+</li>
+<li>
+<p><code>tinker allan 0</code></p>
+</li>
+</ul>
+</div>
+</li>
+</ul>
+</div>
+</li>
+</ul>
+</div>
+<div class="admonitionblock important">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-important" title="Important"></i>
+</td>
+<td class="content">
+The above parameters minimize <code>maximum error</code> at the expense of <code>estimated error</code>,
+the latter might be orders of magnitude above it&#8217;s "normal" value. These parameters also
+may place a greater load on the time server, since they make the servers poll much more
+frequently.
+</td>
+</tr>
+</table>
+</div>
+<div class="ulist bibliography">
+<div class="title">References</div>
+<ul class="bibliography">
+<li>
+<p><a id="1"></a>[1] David Alves, Todd Lipcon and Vijay Garg. Technical Report: HybridTime - Accessible Global Consistency with High Clock Uncertainty. April, 2014. <a href="http://pdsl.ece.utexas.edu/david/hybrid-time-tech-report-01.pdf" class="bare">http://pdsl.ece.utexas.edu/david/hybrid-time-tech-report-01.pdf</a></p>
+</li>
+<li>
+<p><a id="2"></a>[2] James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, J. J. Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang, and Dale Woodford. 2012. Spanner: Google&#8217;s globally-distributed database. In Proceedings of the 10th USENIX conference on Operating Systems Design and Implementation (OSDI'12). USENIX Association, Berkeley, CA, USA, 251-264.</p>
+</li>
+<li>
+<p><a id="3"></a>[3] Alexander Thomson, Thaddeus Diamond, Shu-Chun Weng, Kun Ren, Philip Shao, and Daniel J. Abadi. 2012. Calvin: fast distributed transactions for partitioned database systems. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD '12). ACM, New York, NY, USA, 1-12. DOI=10.1145/2213836.2213838 <a href="http://doi.acm.org/10.1145/2213836.2213838" class="bare">http://doi.acm.org/10.1145/2213836.2213838</a></p>
+</li>
+<li>
+<p><a id="4"></a>[4] Diego Ongaro and John Ousterhout. 2014. In search of an understandable consensus algorithm. In Proceedings of the 2014 USENIX conference on USENIX Annual Technical Conference (USENIX ATC'14), Garth Gibson and Nickolai Zeldovich (Eds.). USENIX Association, Berkeley, CA, USA, 305-320.</p>
+</li>
+<li>
+<p><a id="5"></a>[5] Kwei-Jay Lin, "Consistency issues in real-time database systems," in System Sciences, 1989. Vol.II: Software Track, Proceedings of the Twenty-Second Annual Hawaii International Conference on , vol.2, no., pp.654-661 vol.2, 3-6 Jan 1989 doi: 10.1109/HICSS.1989.48069</p>
+</li>
+</ul>
+</div>
+</div>
+</div>
+    </div>
+    <div class="col-md-3">
+
+  <div id="toc" data-spy="affix" data-offset-top="70">
+  <ul>
+
+      <li>
+
+          <a href="introduction.html">Introducing Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="release_notes.html">Kudu Release Notes</a> 
+      </li> 
+      <li>
+
+          <a href="quickstart.html">Getting Started with Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="installation.html">Installation Guide</a> 
+      </li> 
+      <li>
+
+          <a href="configuration.html">Configuring Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="kudu_impala_integration.html">Using Impala with Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="administration.html">Administering Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="troubleshooting.html">Troubleshooting Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="developing.html">Developing Applications with Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="schema_design.html">Kudu Schema Design</a> 
+      </li> 
+      <li>
+<span class="active-toc">Kudu Transaction Semantics</span>
+            <ul class="sectlevel1">
+<li><a href="#_single_tablet_write_operations">Single tablet write operations</a></li>
+<li><a href="#_writing_to_multiple_tablets">Writing to multiple tablets</a></li>
+<li><a href="#_read_operations_scans">Read Operations (Scans)</a></li>
+<li><a href="#known_issues">Known Issues and Limitations</a></li>
+<li><a href="#_serialization">Serialization</a>
+<ul class="sectlevel2">
+<li><a href="#known_issues_scans">Scans</a></li>
+<li><a href="#_writes">Writes</a></li>
+</ul>
+</li>
+<li><a href="#recommendations">Recommendations</a></li>
+</ul> 
+      </li> 
+      <li>
+
+          <a href="contributing.html">Contributing to Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="style_guide.html">Kudu Documentation Style Guide</a> 
+      </li> 
+      <li>
+
+          <a href="configuration_reference.html">Kudu Configuration Reference</a> 
+      </li> 
+  </ul>
+  </div>
+    </div>
+  </div>
+</div>
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/c78a2f80/releases/0.8.0/docs/troubleshooting.html
----------------------------------------------------------------------
diff --git a/releases/0.8.0/docs/troubleshooting.html b/releases/0.8.0/docs/troubleshooting.html
new file mode 100644
index 0000000..f10566c
--- /dev/null
+++ b/releases/0.8.0/docs/troubleshooting.html
@@ -0,0 +1,457 @@
+---
+title: Apache Kudu (incubating) Troubleshooting
+layout: default
+active_nav: docs
+last_updated: 'Last updated 2016-04-11 08:31:49 PDT'
+---
+<!--
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+
+<div class="container">
+  <div class="row">
+    <div class="col-md-9">
+
+<h1>Apache Kudu (incubating) Troubleshooting</h1>
+      <div class="sect1">
+<h2 id="_issues_starting_the_master_or_tablet_server"><a class="link" href="#_issues_starting_the_master_or_tablet_server">Issues starting the Master or Tablet Server</a></h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="req_hole_punching"><a class="link" href="#req_hole_punching">Errors During Hole Punching Test</a></h3>
+<div class="paragraph">
+<p>Kudu requires hole punching capabilities in order to be efficient. Hole punching support
+depends upon your operation system kernel version and local filesystem implementation.</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>RHEL or CentOS 6.4 or later, patched to kernel version of 2.6.32-358 or later.
+Unpatched RHEL or CentOS 6.4 does not include a kernel with support for hole punching.</p>
+</li>
+<li>
+<p>Ubuntu 14.04 includes version 3.13 of the Linux kernel, which supports hole punching.</p>
+</li>
+<li>
+<p>Newer versions of the EXT4 or XFS file systems support hole punching, but EXT3 does
+not. Older versions of XFS that do not support hole punching return a <code>EOPNOTSUPP</code>
+(operation not supported) error. Older versions of either EXT4 or XFS that do
+not support hole punching cause Kudu to emit an error message such as the following
+and to fail to start:</p>
+<div class="listingblock">
+<div class="content">
+<pre>Error during hole punch test. The log block manager requires a
+filesystem with hole punching support such as ext4 or xfs. On el6,
+kernel version 2.6.32-358 or newer is required. To run without hole
+punching (at the cost of some efficiency and scalability), reconfigure
+Kudu with --block_manager=file. Refer to the Kudu documentation for more
+details. Raw error message follows.</pre>
+</div>
+</div>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>Without hole punching support, the log block manager is unsafe to use. It won&#8217;t
+ever delete blocks, and will consume ever more space on disk.</p>
+</div>
+<div class="paragraph">
+<p>If you can&#8217;t use hole punching in your environment, you can still
+try Kudu. Enable the file block manager instead of the log block manager by
+adding the <code>--block_manager=file</code> flag to the commands you use to start the master
+and tablet servers. The file block manager does not scale as well as the log block
+manager.</p>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+The file block manager is known to scale and perform poorly, and should
+only be used for small-scale evaluation and development.
+</td>
+</tr>
+</table>
+</div>
+</div>
+<div class="sect2">
+<h3 id="ntp"><a class="link" href="#ntp">NTP clock synchronization</a></h3>
+<div class="paragraph">
+<p>For the master and tablet server daemons, the server&#8217;s clock must be synchronized using NTP.
+In addition, the <strong>maximum clock error</strong> (not to be mistaken with the estimated error)
+be below a configurable threshold. The default value is 10 seconds, but it can be set with the flag
+<code>--max_clock_sync_error_usec</code>.</p>
+</div>
+<div class="paragraph">
+<p>If NTP is not installed, or if the clock is reported as unsynchronized, Kudu will not
+start, and will emit a message such as:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>F0924 20:24:36.336809 14550 hybrid_clock.cc:191 Couldn't get the current time: Clock unsynchronized. Status: Service unavailable: Error reading clock. Clock considered unsynchronized.</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>If NTP is installed and synchronized, but the maximum clock error is too high,
+the user will see a message such as:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>Sep 17, 8:13:09.873 PM FATAL hybrid_clock.cc:196 Couldn't get the current time: Clock synchronized, but error: 11130000, is past the maximum allowable error: 10000000</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>or</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>Sep 17, 8:32:31.135 PM FATAL tablet_server_main.cc:38 Check failed: _s.ok() Bad status: Service unavailable: Cannot initialize clock: Cannot initialize HybridClock. Clock synchronized but error was too high (11711000 us).</pre>
+</div>
+</div>
+<div class="admonitionblock tip">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-tip" title="Tip"></i>
+</td>
+<td class="content">
+If NTP is installed the user can monitor the synchronization status by running
+<code>ntptime</code>. The relevant value is what is reported for <code>maximum error</code>.
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>To install NTP, use the appropriate command for your operating system:</p>
+</div>
+<table class="tableblock frame-all grid-all spread">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">OS</th>
+<th class="tableblock halign-left valign-top">Command</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Debian/Ubuntu</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>sudo apt-get install ntp</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">RHEL/CentOS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>sudo yum install ntp</code></p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>If NTP is installed but not running, start it using one of these commands:</p>
+</div>
+<table class="tableblock frame-all grid-all spread">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">OS</th>
+<th class="tableblock halign-left valign-top">Command</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Debian/Ubuntu</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>sudo service ntp restart</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">RHEL/CentOS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>sudo /etc/init.d/ntpd restart</code></p></td>
+</tr>
+</tbody>
+</table>
+<div class="admonitionblock tip">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-tip" title="Tip"></i>
+</td>
+<td class="content">
+NTP requires a network connection and may take a few minutes to synchronize the clock.
+In some cases a spotty network connection may make NTP report the clock as unsynchronized.
+A common, though temporary, workaround for this is to restart NTP with one of the commands above.
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>If the clock is being reported as synchronized by NTP, but the maximum error is too high,
+the user can increase the threshold to a higher value by setting the above
+mentioned flag. For example to increase the possible maximum error to
+20 seconds the flag should be set like: <code>--max_clock_sync_error_usec=20000000</code></p>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_troubleshooting_performance_issues"><a class="link" href="#_troubleshooting_performance_issues">Troubleshooting Performance Issues</a></h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="kudu_tracing"><a class="link" href="#kudu_tracing">Kudu Tracing</a></h3>
+<div class="paragraph">
+<p>The <code>kudu-master</code> and <code>kudu-tserver</code> daemons include built-in tracing support
+based on the open source
+<a href="https://www.chromium.org/developers/how-tos/trace-event-profiling-tool">Chromium Tracing</a>
+framework. You can use tracing to help diagnose latency issues or other problems
+on Kudu servers.</p>
+</div>
+<div class="sect3">
+<h4 id="_accessing_the_tracing_interface"><a class="link" href="#_accessing_the_tracing_interface">Accessing the tracing interface</a></h4>
+<div class="paragraph">
+<p>The tracing interface is accessed via a web browser as part of the
+embedded web server in each of the Kudu daemons.</p>
+</div>
+<table class="tableblock frame-all grid-all spread">
+<caption class="title">Table 1. Tracing Interface URLs</caption>
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Daemon</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">URL</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Tablet Server</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="http://tablet-server-1.example.com:8050/tracing.html" class="bare">http://tablet-server-1.example.com:8050/tracing.html</a></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Master</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="http://master-1.example.com:8051/tracing.html" class="bare">http://master-1.example.com:8051/tracing.html</a></p></td>
+</tr>
+</tbody>
+</table>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+The tracing interface is known to work in recent versions of Google Chrome.
+Other browsers may not work as expected.
+</td>
+</tr>
+</table>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_collecting_a_trace"><a class="link" href="#_collecting_a_trace">Collecting a trace</a></h4>
+<div class="paragraph">
+<p>After navigating to the tracing interface, click the <strong>Record</strong> button on the top left corner
+of the screen. When beginning to diagnose a problem, start by selecting all categories.
+Click <strong>Record</strong> to begin recording a trace.</p>
+</div>
+<div class="paragraph">
+<p>During the trace collection, events are collected into an in-memory ring buffer.
+This ring buffer is fixed in size, so it will eventually fill up to 100%. However, new events
+are still being collected while older events are being removed. While recording the trace,
+trigger the behavior or workload you are interested in exploring.</p>
+</div>
+<div class="paragraph">
+<p>After collecting for several seconds, click <strong>Stop</strong>. The collected trace will be
+downloaded and displayed. Use the <strong>?</strong> key to display help text about using the tracing
+interface to explore the trace.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_saving_a_trace"><a class="link" href="#_saving_a_trace">Saving a trace</a></h4>
+<div class="paragraph">
+<p>You can save collected traces as JSON files for later analysis by clicking <strong>Save</strong>
+after collecting the trace. To load and analyze a saved JSON file, click <strong>Load</strong>
+and choose the file.</p>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_rpc_timeout_traces"><a class="link" href="#_rpc_timeout_traces">RPC Timeout Traces</a></h3>
+<div class="paragraph">
+<p>If client applications are experiencing RPC timeouts, the Kudu tablet server
+<code>WARNING</code> level logs should contain a log entry which includes an RPC-level trace. For example:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>W0922 00:56:52.313848 10858 inbound_call.cc:193] Call kudu.consensus.ConsensusService.UpdateConsensus
+from 192.168.1.102:43499 (request call id 3555909) took 1464ms (client timeout 1000).
+W0922 00:56:52.314888 10858 inbound_call.cc:197] Trace:
+0922 00:56:50.849505 (+     0us) service_pool.cc:97] Inserting onto call queue
+0922 00:56:50.849527 (+    22us) service_pool.cc:158] Handling call
+0922 00:56:50.849574 (+    47us) raft_consensus.cc:1008] Updating replica for 2 ops
+0922 00:56:50.849628 (+    54us) raft_consensus.cc:1050] Early marking committed up to term: 8 index: 880241
+0922 00:56:50.849968 (+   340us) raft_consensus.cc:1056] Triggering prepare for 2 ops
+0922 00:56:50.850119 (+   151us) log.cc:420] Serialized 1555 byte log entry
+0922 00:56:50.850213 (+    94us) raft_consensus.cc:1131] Marking committed up to term: 8 index: 880241
+0922 00:56:50.850218 (+     5us) raft_consensus.cc:1148] Updating last received op as term: 8 index: 880243
+0922 00:56:50.850219 (+     1us) raft_consensus.cc:1195] Filling consensus response to leader.
+0922 00:56:50.850221 (+     2us) raft_consensus.cc:1169] Waiting on the replicates to finish logging
+0922 00:56:52.313763 (+1463542us) raft_consensus.cc:1182] finished
+0922 00:56:52.313764 (+     1us) raft_consensus.cc:1190] UpdateReplicas() finished
+0922 00:56:52.313788 (+    24us) inbound_call.cc:114] Queueing success response</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>These traces can give an indication of which part of the request was slow. Please
+include them in bug reports related to RPC latency outliers.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_kernel_stack_watchdog_traces"><a class="link" href="#_kernel_stack_watchdog_traces">Kernel Stack Watchdog Traces</a></h3>
+<div class="paragraph">
+<p>Each Kudu server process has a background thread called the Stack Watchdog, which
+monitors the other threads in the server in case they have blocked for
+longer-than-expected periods of time. These traces can indicate operating system issues
+or bottlenecked storage.</p>
+</div>
+<div class="paragraph">
+<p>When the watchdog thread identifies a case of thread blockage, it logs an entry
+in the <code>WARNING</code> log like the following:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>W0921 23:51:54.306350 10912 kernel_stack_watchdog.cc:111] Thread 10937 stuck at /data/kudu/consensus/log.cc:505 for 537ms:
+Kernel stack:
+[&lt;ffffffffa00b209d&gt;] do_get_write_access+0x29d/0x520 [jbd2]
+[&lt;ffffffffa00b2471&gt;] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
+[&lt;ffffffffa00fe6d8&gt;] __ext4_journal_get_write_access+0x38/0x80 [ext4]
+[&lt;ffffffffa00d9b23&gt;] ext4_reserve_inode_write+0x73/0xa0 [ext4]
+[&lt;ffffffffa00d9b9c&gt;] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
+[&lt;ffffffffa00d9e90&gt;] ext4_dirty_inode+0x40/0x60 [ext4]
+[&lt;ffffffff811ac48b&gt;] __mark_inode_dirty+0x3b/0x160
+[&lt;ffffffff8119c742&gt;] file_update_time+0xf2/0x170
+[&lt;ffffffff8111c1e0&gt;] __generic_file_aio_write+0x230/0x490
+[&lt;ffffffff8111c4c8&gt;] generic_file_aio_write+0x88/0x100
+[&lt;ffffffffa00d3fb1&gt;] ext4_file_write+0x61/0x1e0 [ext4]
+[&lt;ffffffff81180f5b&gt;] do_sync_readv_writev+0xfb/0x140
+[&lt;ffffffff81181ee6&gt;] do_readv_writev+0xd6/0x1f0
+[&lt;ffffffff81182046&gt;] vfs_writev+0x46/0x60
+[&lt;ffffffff81182102&gt;] sys_pwritev+0xa2/0xc0
+[&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b
+[&lt;ffffffffffffffff&gt;] 0xffffffffffffffff
+
+User stack:
+    @       0x3a1ace10c4  (unknown)
+    @          0x1262103  (unknown)
+    @          0x12622d4  (unknown)
+    @          0x12603df  (unknown)
+    @           0x8e7bfb  (unknown)
+    @           0x8f478b  (unknown)
+    @           0x8f55db  (unknown)
+    @          0x12a7b6f  (unknown)
+    @       0x3a1b007851  (unknown)
+    @       0x3a1ace894d  (unknown)
+    @              (nil)  (unknown)</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>These traces can be useful for diagnosing root-cause latency issues when they are caused by systems
+below Kudu, such as disk controllers or file systems.</p>
+</div>
+</div>
+</div>
+</div>
+    </div>
+    <div class="col-md-3">
+
+  <div id="toc" data-spy="affix" data-offset-top="70">
+  <ul>
+
+      <li>
+
+          <a href="introduction.html">Introducing Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="release_notes.html">Kudu Release Notes</a> 
+      </li> 
+      <li>
+
+          <a href="quickstart.html">Getting Started with Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="installation.html">Installation Guide</a> 
+      </li> 
+      <li>
+
+          <a href="configuration.html">Configuring Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="kudu_impala_integration.html">Using Impala with Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="administration.html">Administering Kudu</a> 
+      </li> 
+      <li>
+<span class="active-toc">Troubleshooting Kudu</span>
+            <ul class="sectlevel1">
+<li><a href="#_issues_starting_the_master_or_tablet_server">Issues starting the Master or Tablet Server</a>
+<ul class="sectlevel2">
+<li><a href="#req_hole_punching">Errors During Hole Punching Test</a></li>
+<li><a href="#ntp">NTP clock synchronization</a></li>
+</ul>
+</li>
+<li><a href="#_troubleshooting_performance_issues">Troubleshooting Performance Issues</a>
+<ul class="sectlevel2">
+<li><a href="#kudu_tracing">Kudu Tracing</a></li>
+<li><a href="#_rpc_timeout_traces">RPC Timeout Traces</a></li>
+<li><a href="#_kernel_stack_watchdog_traces">Kernel Stack Watchdog Traces</a></li>
+</ul>
+</li>
+</ul> 
+      </li> 
+      <li>
+
+          <a href="developing.html">Developing Applications with Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="schema_design.html">Kudu Schema Design</a> 
+      </li> 
+      <li>
+
+          <a href="transaction_semantics.html">Kudu Transaction Semantics</a> 
+      </li> 
+      <li>
+
+          <a href="contributing.html">Contributing to Kudu</a> 
+      </li> 
+      <li>
+
+          <a href="style_guide.html">Kudu Documentation Style Guide</a> 
+      </li> 
+      <li>
+
+          <a href="configuration_reference.html">Kudu Configuration Reference</a> 
+      </li> 
+  </ul>
+  </div>
+    </div>
+  </div>
+</div>
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/c78a2f80/releases/0.8.0/index.md
----------------------------------------------------------------------
diff --git a/releases/0.8.0/index.md b/releases/0.8.0/index.md
new file mode 100644
index 0000000..85a5958
--- /dev/null
+++ b/releases/0.8.0/index.md
@@ -0,0 +1,45 @@
+---
+title: Apache Kudu (incubating) beta release 0.8.0
+layout: single_col
+active_nav: download
+single_col_extra_classes: releases
+---
+
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+
+-->
+
+## Apache Kudu (incubating) beta release 0.8.0
+
+See the [Kudu 0.8.0 Release Notes](docs/release_notes.html).
+
+Downloads of Kudu 0.8.0 are available in the following formats:
+
+* [Kudu 0.8.0 source tarball](http://www.apache.org/closer.cgi?filename=incubator/kudu/0.8.0/apache-kudu-incubating-0.8.0.tar.gz&action=download)
+  ([SHA1](https://www.apache.org/dist/incubator/kudu/0.8.0/apache-kudu-incubating-0.8.0.tar.gz.sha),
+  [MD5](https://www.apache.org/dist/incubator/kudu/0.8.0/apache-kudu-incubating-0.8.0.tar.gz.md5),
+  [Signature](https://www.apache.org/dist/incubator/kudu/0.8.0/apache-kudu-incubating-0.8.0.tar.gz.asc))
+
+You can use the [KEYS file](https://www.apache.org/dist/incubator/kudu/KEYS) to verify the included GPG signature.
+
+Additional links:
+
+* [Kudu 0.8.0 Documentation](docs/)
+* [Kudu 0.8.0 Java API docs](apidocs/)

http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/c78a2f80/releases/index.md
----------------------------------------------------------------------
diff --git a/releases/index.md b/releases/index.md
index fdd85a9..247d6bf 100644
--- a/releases/index.md
+++ b/releases/index.md
@@ -11,11 +11,11 @@ Kudu is currently in a beta period. It is not yet considered to be production-re
 
 ### Latest beta release
 
-* **[Kudu 0.7.1 (beta)](0.7.1/)** was released on March 9, 2016. It is the latest version of Kudu.
-  It is a bug fix release.
+* **[Kudu 0.8.0 (beta)](0.8.0/)** was released on April 10, 2016.
 
 ### Previous beta releases
 
+* [Kudu 0.7.1 (beta)](0.7.1/) was released on March 9, 2016. It is a bug fix release.
 * [Kudu 0.7.0 (beta)](0.7.0/) was released on Feb 26, 2016.
   It is also the first version to be released as part of incubation in the Apache Software Foundation.
 * [Kudu 0.6.0 (beta)](0.6.0/) was released on Nov 24, 2015.



Mime
View raw message