kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From t...@apache.org
Subject [1/2] kudu-site git commit: Publish commit(s) from site source repo: e8994d4 Add weekly update for 8/16
Date Wed, 17 Aug 2016 00:34:14 GMT
Repository: kudu-site
Updated Branches:
  refs/heads/asf-site 91f92f706 -> 3bcd2b706


http://git-wip-us.apache.org/repos/asf/kudu-site/blob/3bcd2b70/feed.xml
----------------------------------------------------------------------
diff --git a/feed.xml b/feed.xml
index 4cccc32..69c0c7a 100644
--- a/feed.xml
+++ b/feed.xml
@@ -1,4 +1,121 @@
-<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator
uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml"
rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html"
/><updated>2016-08-11T16:45:45-07:00</updated><id>/</id><entry><title>Apache
Kudu Weekly Update August 8th, 2016</title><link href="/2016/08/08/weekly-update.html"
rel="alternate" type="text/html" title="Apache Kudu Weekly Update August 8th, 2016" /><published>2016-08-08T00:00:00-07:00</published><updated>2016-08-08T00:00:00-07:00</updated><id>/2016/08/08/weekly-update</id><content
type="html" xml:base="/2016/08/08/weekly-update.html">&lt;p&gt;Welcome to the nineteenth
edition of the Kudu Weekly Update. This weekly blog post
+<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator
uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml"
rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html"
/><updated>2016-08-16T17:26:09-07:00</updated><id>/</id><entry><title>Apache
Kudu Weekly Update August 16th, 2016</title><link href="/2016/08/16/weekly-update.html"
rel="alternate" type="text/html" title="Apache Kudu Weekly Update August 16th, 2016" /><published>2016-08-16T00:00:00-07:00</published><updated>2016-08-16T00:00:00-07:00</updated><id>/2016/08/16/weekly-update</id><content
type="html" xml:base="/2016/08/16/weekly-update.html">&lt;p&gt;Welcome to the twentieth
edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu project.&lt;/p&gt;
+
+&lt;!--more--&gt;
+
+&lt;h2 id=&quot;project-news&quot;&gt;Project news&lt;/h2&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;
+    &lt;p&gt;The first release candidate for the 0.10.0 is &lt;a href=&quot;http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201608.mbox/%3CCADY20s7U5jVpozFg3L%3DDz2%2B4AenGineJvH96A_HAM12biDjPJA%40mail.gmail.com%3E&quot;&gt;now
available&lt;/a&gt;&lt;/p&gt;
+
+    &lt;p&gt;Community developers and users are encouraged to download the source
+tarball and vote on the release.&lt;/p&gt;
+
+    &lt;p&gt;For information on what’s new, check out the
+&lt;a href=&quot;https://github.com/apache/kudu/blob/master/docs/release_notes.adoc#rn_0.10.0&quot;&gt;release
notes&lt;/a&gt;.
+&lt;em&gt;Note:&lt;/em&gt; some links from these in-progress release notes
will not be live until the
+release itself is published.&lt;/p&gt;
+  &lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;h2 id=&quot;development-discussions-and-code-in-progress&quot;&gt;Development
discussions and code in progress&lt;/h2&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;
+    &lt;p&gt;Will Berkeley spent some time working on the Spark integration this
week
+to add support for UPSERT as well as other operations.
+Dan Burkert pitched in a bit with some &lt;a href=&quot;http://mail-archives.apache.org/mod_mbox/incubator-kudu-dev/201608.mbox/%3CCALo2W-XBoSz9cbhXi81ipubrAYgqyDiEeHz-ys8sPAshfcik6w%40mail.gmail.com%3E&quot;&gt;suggestions&lt;/a&gt;
+which were then integrated in a &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3871/&quot;&gt;patch&lt;/a&gt;
+provided by Will.&lt;/p&gt;
+
+    &lt;p&gt;After some reviews by Dan, Chris George, and Ram Mettu, the patch was
committed
+in time for the upcoming 0.10.0 release.&lt;/p&gt;
+  &lt;/li&gt;
+  &lt;li&gt;
+    &lt;p&gt;Dan Burkert also completed work for the new &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3854/&quot;&gt;manual
partitioning APIs&lt;/a&gt;
+in the Java client. After finishing up the basic implementation, Dan also made some
+cleanups to the related APIs in both the &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3958/&quot;&gt;Java&lt;/a&gt;
+and &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3882/&quot;&gt;C++&lt;/a&gt;
clients.&lt;/p&gt;
+
+    &lt;p&gt;Dan and Misty Stanley-Jones also collaborated to finish the
+&lt;a href=&quot;https://gerrit.cloudera.org/#/c/3796/&quot;&gt;documentation&lt;/a&gt;
+for this new feature.&lt;/p&gt;
+  &lt;/li&gt;
+  &lt;li&gt;
+    &lt;p&gt;Adar Dembo worked on some tooling to allow users to migrate their Kudu
clusters
+from a single-master configuration to a multi-master one. Along the way, he
+started building some common infrastructure for command-line tooling.&lt;/p&gt;
+
+    &lt;p&gt;Since Kudu’s initial release, it has included separate binaries for
different
+administrative or operational tools (e.g. &lt;code&gt;kudu-ts-cli&lt;/code&gt;,
&lt;code&gt;kudu-ksck&lt;/code&gt;, &lt;code&gt;kudu-fs_dump&lt;/code&gt;,
+&lt;code&gt;log-dump&lt;/code&gt;, etc). Despite having similar usage, these
tools don’t share much code,
+and the separate statically linked binaries make the Kudu packages take more disk
+space than strictly necessary.&lt;/p&gt;
+
+    &lt;p&gt;Adar’s work has introduced a new top-level &lt;code&gt;kudu&lt;/code&gt;
binary which exposes a set of subcommands,
+much like the &lt;code&gt;git&lt;/code&gt; and &lt;code&gt;docker&lt;/code&gt;
binaries with which readers may be familiar.
+For example, a new tool he has built for dumping peer identifiers from a tablet’s
+consensus metadata is triggered using &lt;code&gt;kudu tablet cmeta print_replica_uuids&lt;/code&gt;.&lt;/p&gt;
+
+    &lt;p&gt;This new tool will be available in the upcoming 0.10.0 release; however,
migration
+of the existing tools to the new infrastructure has not yet been completed. We
+expect that by Kudu 1.0, the old tools will be removed in favor of more subcommands
+of the &lt;code&gt;kudu&lt;/code&gt; tool.&lt;/p&gt;
+  &lt;/li&gt;
+  &lt;li&gt;
+    &lt;p&gt;Todd Lipcon picked up the work started by David Alves in July to provide
+&lt;a href=&quot;https://gerrit.cloudera.org/#/c/2642/&quot;&gt;“exactly-once”
semantics&lt;/a&gt; for write operations.
+Todd carried the patch series through review and also completed integration of the
+feature into the Kudu server processes.&lt;/p&gt;
+
+    &lt;p&gt;After testing the feature for several days on a large cluster under
load,
+the team decided to enable this new feature by default in Kudu 0.10.0.&lt;/p&gt;
+  &lt;/li&gt;
+  &lt;li&gt;
+    &lt;p&gt;Mike Percy resumed working on garbage collection of &lt;a href=&quot;https://gerrit.cloudera.org/#/c/2853/&quot;&gt;past
versions of
+updated and deleted rows&lt;/a&gt;. His &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3076/&quot;&gt;main
+patch for the feature&lt;/a&gt; went through
+several rounds of review and testing, but unfortunately missed the cut-off
+for 0.10.0.&lt;/p&gt;
+  &lt;/li&gt;
+  &lt;li&gt;
+    &lt;p&gt;Alexey Serbin’s work to add doxygen-based documentation for the C++
Client API
+was &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3840/&quot;&gt;committed&lt;/a&gt;
this week. These
+docs will be published as part of the 0.10.0 release.&lt;/p&gt;
+  &lt;/li&gt;
+  &lt;li&gt;
+    &lt;p&gt;Alexey also continued work on implementing the &lt;code&gt;AUTO_FLUSH_BACKGROUND&lt;/code&gt;
write
+mode for the C++ client. This feature makes it easier to implement high-throughput
+ingest using the C++ API by automatically handling the batching and flushing of writes
+based on a configurable buffer size.&lt;/p&gt;
+
+    &lt;p&gt;Alexey’s &lt;a href=&quot;https://gerrit.cloudera.org/#/c/3952/&quot;&gt;patch&lt;/a&gt;
has received several
+rounds of review and looks likely to be committed soon. Detailed performance testing
+will follow.&lt;/p&gt;
+  &lt;/li&gt;
+  &lt;li&gt;
+    &lt;p&gt;Congratulations to Ram Mettu for committing his first patch to Kudu
this week!
+Ram fixed a &lt;a href=&quot;https://issues.apache.org/jira/browse/KUDU-1522&quot;&gt;bug
in handling Alter Table with TIMESTAMP columns&lt;/a&gt;.&lt;/p&gt;
+  &lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;h2 id=&quot;upcoming-talks&quot;&gt;Upcoming talks&lt;/h2&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;Mike Percy will be speaking about Kudu this Wednesday at the
+&lt;a href=&quot;http://www.meetup.com/Denver-Cloudera-User-Group/events/232782782/&quot;&gt;Denver
Cloudera User Group&lt;/a&gt;
+and on Thursday at the
+&lt;a href=&quot;http://www.meetup.com/Boulder-Denver-Big-Data/events/232056701/&quot;&gt;Boulder/Denver
Big Data Meetup&lt;/a&gt;.
+If you’re based in the Boulder/Denver area, be sure not to miss these talks!&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;p&gt;Want to learn more about a specific topic from this blog post? Shoot an
email to the
+&lt;a href=&quot;&amp;#109;&amp;#097;&amp;#105;&amp;#108;&amp;#116;&amp;#111;:&amp;#117;&amp;#115;&amp;#101;&amp;#114;&amp;#064;&amp;#107;&amp;#117;&amp;#100;&amp;#117;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&quot;&gt;kudu-user
mailing list&lt;/a&gt; or
+tweet at &lt;a href=&quot;https://twitter.com/ApacheKudu&quot;&gt;@ApacheKudu&lt;/a&gt;.
Similarly, if you’re
+aware of some Kudu news we missed, let us know so we can cover it in
+a future post.&lt;/p&gt;</content><author><name>Todd Lipcon</name></author><summary>Welcome
to the twentieth edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu project.</summary></entry><entry><title>Apache
Kudu Weekly Update August 8th, 2016</title><link href="/2016/08/08/weekly-update.html"
rel="alternate" type="text/html" title="Apache Kudu Weekly Update August 8th, 2016" /><published>2016-08-08T00:00:00-07:00</published><updated>2016-08-08T00:00:00-07:00</updated><id>/2016/08/08/weekly-update</id><content
type="html" xml:base="/2016/08/08/weekly-update.html">&lt;p&gt;Welcome to the nineteenth
edition of the Kudu Weekly Update. This weekly blog post
 covers ongoing development and news in the Apache Kudu project.&lt;/p&gt;
 
 &lt;!--more--&gt;
@@ -519,7 +636,7 @@ three-node (or five-node) deployment.&lt;/li&gt;
 &lt;a href=&quot;https://issues.apache.org/jira/browse/KUDU-1474&quot;&gt;KUDU-1474&lt;/a&gt;,
and there’s been
 &lt;a href=&quot;http://gerrit.cloudera.org:8080/3393&quot;&gt;some discussion&lt;/a&gt;
around a design, but
 nothing has been implemented yet. Stay tuned!&lt;/p&gt;</content><author><name>Adar
Dembo</name></author><summary>This blog post describes how the 1.0 release
of Apache Kudu (incubating) will
-support fault tolerance for the Kudu master, finally eliminating Kudu’s last
+support fault tolerance for the Kudu master, finally eliminating Kudu&amp;#8217;s last
 single point of failure.</summary></entry><entry><title>Apache Kudu
(incubating) Weekly Update June 21, 2016</title><link href="/2016/06/21/weekly-update.html"
rel="alternate" type="text/html" title="Apache Kudu (incubating) Weekly Update June 21, 2016"
/><published>2016-06-21T00:00:00-07:00</published><updated>2016-06-21T00:00:00-07:00</updated><id>/2016/06/21/weekly-update</id><content
type="html" xml:base="/2016/06/21/weekly-update.html">&lt;p&gt;Welcome to the fourteenth
edition of the Kudu Weekly Update. This weekly blog post
 covers ongoing development and news in the Apache Kudu (incubating) project.&lt;/p&gt;
 
@@ -558,94 +675,4 @@ uses Raft consensus on a single node, and some changes we’re making
as Kudu is
 tweet at &lt;a href=&quot;https://twitter.com/ApacheKudu&quot;&gt;@ApacheKudu&lt;/a&gt;.
Similarly, if you’re
 aware of some Kudu news we missed, let us know so we can cover it in
 a future post.&lt;/p&gt;</content><author><name>Jean-Daniel Cryans</name></author><summary>Welcome
to the fourteenth edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry><entry><title>Using
Raft Consensus on a Single Node</title><link href="/2016/06/17/raft-consensus-single-node.html"
rel="alternate" type="text/html" title="Using Raft Consensus on a Single Node" /><published>2016-06-17T00:00:00-07:00</published><updated>2016-06-17T00:00:00-07:00</updated><id>/2016/06/17/raft-consensus-single-node</id><content
type="html" xml:base="/2016/06/17/raft-consensus-single-node.html">&lt;p&gt;As
Kudu marches toward its 1.0 release, which will include support for
-multi-master operation, we are working on removing old code that is no longer
-needed. One such piece of code is called LocalConsensus. Once LocalConsensus is
-removed, we will be using Raft consensus even on Kudu tables that have a
-replication factor of 1.&lt;/p&gt;
-
-&lt;!--more--&gt;
-
-&lt;p&gt;Using Raft consensus in single-node cases is important for multi-master
-support because it will allow people to dynamically increase their Kudu
-cluster’s existing master server replication factor from 1 to many (3 or 5 are
-typical).&lt;/p&gt;
-
-&lt;h1 id=&quot;the-consensus-interface&quot;&gt;The Consensus interface&lt;/h1&gt;
-
-&lt;p&gt;In Kudu, the
-&lt;a href=&quot;https://github.com/apache/incubator-kudu/blob/branch-0.9.x/src/kudu/consensus/consensus.h&quot;&gt;Consensus&lt;/a&gt;
-interface was created as an abstraction to allow us to build the plumbing
-around how a consensus implementation would interact with the underlying
-tablet. We were able to build out this “scaffolding” long before our Raft
-implementation was complete.&lt;/p&gt;
-
-&lt;p&gt;The Consensus API has the following main responsibilities:&lt;/p&gt;
-
-&lt;ol&gt;
-  &lt;li&gt;Support acting as a Raft &lt;code&gt;LEADER&lt;/code&gt;
and replicate writes to a local
-write-ahead log (WAL) as well as followers in the Raft configuration. For
-each operation written to the leader, a Raft implementation must keep track
-of how many nodes have written a copy of the operation being replicated, and
-whether or not that constitutes a majority. Once a majority of the nodes
-have written a copy of the data, it is considered committed.&lt;/li&gt;
-  &lt;li&gt;Support acting as a Raft &lt;code&gt;FOLLOWER&lt;/code&gt;
by accepting writes from the leader and
-preparing them to be eventually committed.&lt;/li&gt;
-  &lt;li&gt;Support voting in and initiating leader elections.&lt;/li&gt;
-  &lt;li&gt;Support participating in and initiating configuration changes (such as
going
-from a replication factor of 3 to 4).&lt;/li&gt;
-&lt;/ol&gt;
-
-&lt;p&gt;The first implementation of the Consensus interface was called LocalConsensus.
-LocalConsensus only supported acting as a leader of a single-node configuration
-(hence the name “local”). It could not replicate to followers, participate in
-elections, or change configurations. These limitations have led us to
-&lt;a href=&quot;https://gerrit.cloudera.org/3350&quot;&gt;remove&lt;/a&gt;
LocalConsensus from the code base
-entirely.&lt;/p&gt;
-
-&lt;p&gt;Because Kudu has a full-featured Raft implementation, Kudu’s RaftConsensus
-supports all of the above functions of the Consensus interface.&lt;/p&gt;
-
-&lt;h1 id=&quot;using-a-single-node-raft-configuration&quot;&gt;Using a Single-node
Raft configuration&lt;/h1&gt;
-
-&lt;p&gt;A common question on the Raft mailing lists is: “Is it even possible to
use
-Raft on a single node?” The answer is yes.&lt;/p&gt;
-
-&lt;p&gt;Fundamentally, Raft works by first electing a leader that is responsible
for
-replicating write operations to the other members of the configuration. In
-order to elect a leader, Raft requires a (strict) majority of the voters to
-vote “yes” in an election. When there is only a single eligible node in the
-configuration, there is no chance of losing the election. Raft specifies that
-when starting an election, a node must first vote for itself and then contact
-the rest of the voters to tally their votes. If there is only a single node, no
-communication is required and an election succeeds instantaneously.&lt;/p&gt;
-
-&lt;p&gt;So, when does it make sense to use Raft for a single node?&lt;/p&gt;
-
-&lt;p&gt;It makes sense to do this when you want to allow growing the replication
factor
-in the future. This is something that Kudu needs to support. When deploying
-Kudu, someone may wish to test it out with limited resources in a small
-environment. Eventually, they may wish to transition that cluster to be a
-staging or production environment, which would typically require the fault
-tolerance achievable with multi-node Raft. Without a consensus implementation
-that supports configuration changes, there would be no way to gracefully
-support this. Because single-node Raft supports dynamically adding an
-additional node to its configuration, it is possible to go from one replica to
-2 and then 3 replicas and end up with a fault-tolerant cluster without
-incurring downtime.&lt;/p&gt;
-
-&lt;h1 id=&quot;more-about-raft&quot;&gt;More about Raft&lt;/h1&gt;
-
-&lt;p&gt;To learn more about how Kudu uses Raft consensus, you may find the relevant
-&lt;a href=&quot;https://github.com/apache/incubator-kudu/blob/master/docs/design-docs/README.md&quot;&gt;design
docs&lt;/a&gt;
-interesting. In the future, we may also post more articles on the Kudu blog
-about how Kudu uses Raft to achieve fault tolerance.&lt;/p&gt;
-
-&lt;p&gt;To learn more about the Raft protocol itself, please see the &lt;a href=&quot;https://raft.github.io/&quot;&gt;Raft
consensus
-home page&lt;/a&gt;. The design of Kudu’s Raft implementation
-is based on the extended protocol described in Diego Ongaro’s Ph.D.
-dissertation, which you can find linked from the above web site.&lt;/p&gt;</content><author><name>Mike
Percy</name></author><summary>As Kudu marches toward its 1.0 release, which
will include support for
-multi-master operation, we are working on removing old code that is no longer
-needed. One such piece of code is called LocalConsensus. Once LocalConsensus is
-removed, we will be using Raft consensus even on Kudu tables that have a
-replication factor of 1.</summary></entry></feed>
+covers ongoing development and news in the Apache Kudu (incubating) project.</summary></entry></feed>


Mime
View raw message