From commits-return-6449-archive-asf-public=cust-asf.ponee.io@kudu.apache.org Wed Sep 26 19:56:40 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 49FC818077F for ; Wed, 26 Sep 2018 19:56:38 +0200 (CEST) Received: (qmail 36219 invoked by uid 500); 26 Sep 2018 17:56:37 -0000 Mailing-List: contact commits-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kudu.apache.org Delivered-To: mailing list commits@kudu.apache.org Received: (qmail 36103 invoked by uid 99); 26 Sep 2018 17:56:37 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Sep 2018 17:56:37 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id C132EE0630; Wed, 26 Sep 2018 17:56:36 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: mpercy@apache.org To: commits@kudu.apache.org Date: Wed, 26 Sep 2018 17:56:38 -0000 Message-Id: <434f149ff03745f880c70ef250785f1a@git.apache.org> In-Reply-To: <8410d872324940918a1a19b01d8825af@git.apache.org> References: <8410d872324940918a1a19b01d8825af@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [3/4] kudu-site git commit: Publish commit(s) from site source repo: 83530755d Blogpost describing index skip scan optimization. http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/08/31/intro-flume-kudu-sink.html ---------------------------------------------------------------------- diff --git a/2016/08/31/intro-flume-kudu-sink.html b/2016/08/31/intro-flume-kudu-sink.html index 4bf4f7d..7f94bc9 100644 --- a/2016/08/31/intro-flume-kudu-sink.html +++ b/2016/08/31/intro-flume-kudu-sink.html @@ -225,7 +225,7 @@ release and the source code can be found agent1.sources = source1 agent1.channels = channel1 agent1.sinks = sink1 @@ -243,25 +243,25 @@ agent1.sinks.sink1.tableName = stats agent1.sinks.sink1.channel = channel1 agent1.sinks.sink1.batchSize = 50 agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer - + -

We define a source called source1 which simply executes a vmstat command to continuously generate -virtual memory statistics for the machine and queue events into an in-memory channel1 channel, -which in turn is used for writing these events to a Kudu table called stats. We are using -org.apache.kudu.flume.sink.SimpleKuduEventProducer as the producer. SimpleKuduEventProducer is +

We define a source called source1 which simply executes a vmstat command to continuously generate +virtual memory statistics for the machine and queue events into an in-memory channel1 channel, +which in turn is used for writing these events to a Kudu table called stats. We are using +org.apache.kudu.flume.sink.SimpleKuduEventProducer as the producer. SimpleKuduEventProducer is the built-in and default producer, but it’s implemented as a showcase for how to write Flume events into Kudu tables. For any serious functionality we’d have to write a custom producer. We -need to make this producer and the KuduSink class available to Flume. We can do that by simply -copying the kudu-flume-sink-<VERSION>.jar jar file from the Kudu distribution to the -$FLUME_HOME/plugins.d/kudu-sink/lib directory in the Flume installation. The jar file contains -KuduSink and all of its dependencies (including Kudu java client classes).

+need to make this producer and the KuduSink class available to Flume. We can do that by simply +copying the kudu-flume-sink-<VERSION>.jar jar file from the Kudu distribution to the +$FLUME_HOME/plugins.d/kudu-sink/lib directory in the Flume installation. The jar file contains +KuduSink and all of its dependencies (including Kudu java client classes).

At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are -(agent1.sinks.sink1.masterAddresses = localhost) and which Kudu table should be used for writing -Flume events to (agent1.sinks.sink1.tableName = stats). The Kudu Flume Sink doesn’t create this +(agent1.sinks.sink1.masterAddresses = localhost) and which Kudu table should be used for writing +Flume events to (agent1.sinks.sink1.tableName = stats). The Kudu Flume Sink doesn’t create this table, it has to be created before the Kudu Flume Sink is started.

-

You may also notice the batchSize parameter. Batch size is used for batching up to that many +

You may also notice the batchSize parameter. Batch size is used for batching up to that many Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge impact on ingest performance of the Kudu cluster.

@@ -311,89 +311,89 @@ impact on ingest performance of the Kudu cluster.

Let’s take a look at the source code for the built-in producer class:

-
public class SimpleKuduEventProducer implements KuduEventProducer {
-  private byte[] payload;
-  private KuduTable table;
-  private String payloadColumn;
-
-  public SimpleKuduEventProducer(){
-  }
-
-  @Override
-  public void configure(Context context) {
-    payloadColumn = context.getString("payloadColumn","payload");
-  }
-
-  @Override
-  public void configure(ComponentConfiguration conf) {
-  }
-
-  @Override
-  public void initialize(Event event, KuduTable table) {
-    this.payload = event.getBody();
-    this.table = table;
-  }
-
-  @Override
-  public List<Operation> getOperations() throws FlumeException {
-    try {
-      Insert insert = table.newInsert();
-      PartialRow row = insert.getRow();
-      row.addBinary(payloadColumn, payload);
-
-      return Collections.singletonList((Operation) insert);
-    } catch (Exception e){
-      throw new FlumeException("Failed to create Kudu Insert object!", e);
-    }
-  }
-
-  @Override
-  public void close() {
-  }
-}
-
- -

SimpleKuduEventProducer implements the org.apache.kudu.flume.sink.KuduEventProducer interface, +

public class SimpleKuduEventProducer implements KuduEventProducer { + private byte[] payload; + private KuduTable table; + private String payloadColumn; + + public SimpleKuduEventProducer(){ + } + + @Override + public void configure(Context context) { + payloadColumn = context.getString("payloadColumn","payload"); + } + + @Override + public void configure(ComponentConfiguration conf) { + } + + @Override + public void initialize(Event event, KuduTable table) { + this.payload = event.getBody(); + this.table = table; + } + + @Override + public List<Operation> getOperations() throws FlumeException { + try { + Insert insert = table.newInsert(); + PartialRow row = insert.getRow(); + row.addBinary(payloadColumn, payload); + + return Collections.singletonList((Operation) insert); + } catch (Exception e){ + throw new FlumeException("Failed to create Kudu Insert object!", e); + } + } + + @Override + public void close() { + } +} +
+ +

SimpleKuduEventProducer implements the org.apache.kudu.flume.sink.KuduEventProducer interface, which itself looks like this:

-
public interface KuduEventProducer extends Configurable, ConfigurableComponent {
-  /**
+
public interface KuduEventProducer extends Configurable, ConfigurableComponent {
+  /**
    * Initialize the event producer.
    * @param event to be written to Kudu
    * @param table the KuduTable object used for creating Kudu Operation objects
-   */
-  void initialize(Event event, KuduTable table);
+   */
+  void initialize(Event event, KuduTable table);
 
-  /**
+  /**
    * Get the operations that should be written out to Kudu as a result of this
    * event. This list is written to Kudu using the Kudu client API.
    * @return List of {@link org.kududb.client.Operation} which
    * are written as such to Kudu
-   */
-  List<Operation> getOperations();
+   */
+  List<Operation> getOperations();
 
-  /*
+  /*
    * Clean up any state. This will be called when the sink is being stopped.
-   */
-  void close();
-}
-
+ */ + void close(); +} + -

public void configure(Context context) is called when an instance of our producer is instantiated +

public void configure(Context context) is called when an instance of our producer is instantiated by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named -payloadColumn and uses its value (“payload” if not overridden in Flume configuration file) as the +payloadColumn and uses its value (“payload” if not overridden in Flume configuration file) as the column which will hold the value of the Flume event payload. If you recall from above, we had -configured the KuduSink to listen for events generated from the vmstat command. Each output row -from that command will be stored as a new row containing a payload column in the stats table. -SimpleKuduEventProducer does not have any configuration parameters, but if it had any we would -define them by prefixing it with producer. (agent1.sinks.sink1.producer.parameter1 for +configured the KuduSink to listen for events generated from the vmstat command. Each output row +from that command will be stored as a new row containing a payload column in the stats table. +SimpleKuduEventProducer does not have any configuration parameters, but if it had any we would +define them by prefixing it with producer. (agent1.sinks.sink1.producer.parameter1 for example).

-

The main producer logic resides in the public List<Operation> getOperations() method. In +

The main producer logic resides in the public List<Operation> getOperations() method. In SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into -the Kudu table. Here we call Kudu’s newInsert() to initiate an insert, but could have used -Upsert if updating an existing row was also an option, in fact there’s another producer -implementation available for doing just that: SimpleKeyedKuduEventProducer. Most probably you +the Kudu table. Here we call Kudu’s newInsert() to initiate an insert, but could have used +Upsert if updating an existing row was also an option, in fact there’s another producer +implementation available for doing just that: SimpleKeyedKuduEventProducer. Most probably you will need to write your own custom producer in the real world, but you can base your implementation on the built-in ones.

@@ -423,6 +423,8 @@ is included in the Kudu distribution. You can follow him on Twitter at

Recent posts

 http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/09/16/predicate-pushdown.html ---------------------------------------------------------------------- diff --git a/2016/09/16/predicate-pushdown.html b/2016/09/16/predicate-pushdown.html index 8af7be3..a1c26a2 100644 --- a/2016/09/16/predicate-pushdown.html +++ b/2016/09/16/predicate-pushdown.html @@ -269,6 +269,8 @@ coordinators, and from the Cloudera community as a whole.

Recent posts

http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/09/20/apache-kudu-1-0-0-released.html ---------------------------------------------------------------------- diff --git a/2016/09/20/apache-kudu-1-0-0-released.html b/2016/09/20/apache-kudu-1-0-0-released.html index 5c740fd..c8deda1 100644 --- a/2016/09/20/apache-kudu-1-0-0-released.html +++ b/2016/09/20/apache-kudu-1-0-0-released.html @@ -132,15 +132,15 @@ history of all changes made to a given table since the beginning of time.

  • Most of Kudu’s command line tools have been consolidated under a new -top-level kudu tool. This reduces the number of large binaries distributed +top-level kudu tool. This reduces the number of large binaries distributed with Kudu and also includes much-improved help output.

  • -

    Administrative tools including kudu cluster ksck now support running +

    Administrative tools including kudu cluster ksck now support running against multi-master Kudu clusters.

  • -

    The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND mode. +

    The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND mode. This can provide higher throughput for ingest workloads.

    • @@ -164,6 +164,8 @@ repository.

    Recent posts

    http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/09/26/strata-nyc-kudu-talks.html ---------------------------------------------------------------------- diff --git a/2016/09/26/strata-nyc-kudu-talks.html b/2016/09/26/strata-nyc-kudu-talks.html index 27e8926..e3f79fb 100644 --- a/2016/09/26/strata-nyc-kudu-talks.html +++ b/2016/09/26/strata-nyc-kudu-talks.html @@ -183,6 +183,8 @@ Be sure to RSVP as spots are filling up fast.

    Recent posts

    http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/10/11/weekly-update.html ---------------------------------------------------------------------- diff --git a/2016/10/11/weekly-update.html b/2016/10/11/weekly-update.html index 3ce1988..03d8a34 100644 --- a/2016/10/11/weekly-update.html +++ b/2016/10/11/weekly-update.html @@ -189,14 +189,14 @@ look and leave your comments! Similarly, if you are interested in contributing in any of these areas, please feel free to volunteer on the mailing list. Help of all kinds (coding, documentation, testing, etc) is welcomed.

    -
  • Adar Dembo spent a chunk of time re-working the thirdparty directory +
  • Adar Dembo spent a chunk of time re-working the thirdparty directory that contains most of Kudu’s native dependencies. The major resulting changes are:
    • Build directories are now cleanly isolated from source directories, improving cleanliness of re-builds.
      • -
    • ThreadSanitizer (TSAN) builds now use libc++ instead of libstdcxx -for C++ library support. The libc++ library has better support for +
    • ThreadSanitizer (TSAN) builds now use libc++ instead of libstdcxx +for C++ library support. The libc++ library has better support for sanitizers, is easier to build in isolation, and solves some compatibility issues that Adar was facing with GCC 5 on Ubuntu Xenial.
    • All of the thirdparty dependencies now build with TSAN instrumentation, @@ -236,14 +236,14 @@ to increase.

    • Dan Burkert picked up work originally started by Sameer Abhyankar on KUDU-1363, which adds -support for adding IN (...) predicates to scanners. Dan committed the +support for adding IN (...) predicates to scanners. Dan committed the main patch as well as corresponding support in the Java client. Jordan Birdsell quickly added corresponding support in Python. This new feature will be available in an upcoming release.

    • -

      Work continues on the kudu command line tool. Dinesh Bhat added +

      Work continues on the kudu command line tool. Dinesh Bhat added the ability to ask a tablet’s leader to step down and Alexey Serbin added a tool to insert random data into a table.

      @@ -287,6 +287,8 @@ a future post.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/10/20/weekly-update.html ---------------------------------------------------------------------- diff --git a/2016/10/20/weekly-update.html b/2016/10/20/weekly-update.html index 90f273d..cb641b8 100644 --- a/2016/10/20/weekly-update.html +++ b/2016/10/20/weekly-update.html @@ -216,6 +216,8 @@ a future post.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/11/01/weekly-update.html ---------------------------------------------------------------------- diff --git a/2016/11/01/weekly-update.html b/2016/11/01/weekly-update.html index 7990409..66ee4ae 100644 --- a/2016/11/01/weekly-update.html +++ b/2016/11/01/weekly-update.html @@ -214,6 +214,8 @@ a future post.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/11/15/weekly-update.html ---------------------------------------------------------------------- diff --git a/2016/11/15/weekly-update.html b/2016/11/15/weekly-update.html index 3ad0360..03ae6a3 100644 --- a/2016/11/15/weekly-update.html +++ b/2016/11/15/weekly-update.html @@ -235,6 +235,8 @@ a future post.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/01/20/apache-kudu-1-2-0-released.html ---------------------------------------------------------------------- diff --git a/2017/01/20/apache-kudu-1-2-0-released.html b/2017/01/20/apache-kudu-1-2-0-released.html index 104074f..aa635bb 100644 --- a/2017/01/20/apache-kudu-1-2-0-released.html +++ b/2017/01/20/apache-kudu-1-2-0-released.html @@ -160,6 +160,8 @@ repository.

      • Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/03/20/apache-kudu-1-3-0-released.html ---------------------------------------------------------------------- diff --git a/2017/03/20/apache-kudu-1-3-0-released.html b/2017/03/20/apache-kudu-1-3-0-released.html index 4dc68e5..5ba47db 100644 --- a/2017/03/20/apache-kudu-1-3-0-released.html +++ b/2017/03/20/apache-kudu-1-3-0-released.html @@ -159,6 +159,8 @@ repository.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/04/19/apache-kudu-1-3-1-released.html ---------------------------------------------------------------------- diff --git a/2017/04/19/apache-kudu-1-3-1-released.html b/2017/04/19/apache-kudu-1-3-1-released.html index fcd206f..4482d7a 100644 --- a/2017/04/19/apache-kudu-1-3-1-released.html +++ b/2017/04/19/apache-kudu-1-3-1-released.html @@ -143,6 +143,8 @@ repository.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/06/13/apache-kudu-1-4-0-released.html ---------------------------------------------------------------------- diff --git a/2017/06/13/apache-kudu-1-4-0-released.html b/2017/06/13/apache-kudu-1-4-0-released.html index b484adb..6e1a205 100644 --- a/2017/06/13/apache-kudu-1-4-0-released.html +++ b/2017/06/13/apache-kudu-1-4-0-released.html @@ -132,7 +132,7 @@ improvements, optimizations, and bug fixes.

    • a new C++ client API to efficiently map primary keys to their associated partitions and hosts
    • support for long-running fault-tolerant scans in the Java client
      • -
    • a new kudu fs check command which can perform offline consistency checks +
    • a new kudu fs check command which can perform offline consistency checks and repairs on the local on-disk storage of a Tablet Server or Master.
    • many optimizations to reduce disk space usage, improve write throughput, and improve throughput of background maintenance operations.
      • @@ -159,6 +159,8 @@ repository.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/09/08/apache-kudu-1-5-0-released.html ---------------------------------------------------------------------- diff --git a/2017/09/08/apache-kudu-1-5-0-released.html b/2017/09/08/apache-kudu-1-5-0-released.html index dccee96..df182a4 100644 --- a/2017/09/08/apache-kudu-1-5-0-released.html +++ b/2017/09/08/apache-kudu-1-5-0-released.html @@ -136,9 +136,9 @@ scenarios additional reductions planned for the future
    • a new configuration dashboard on the web UI which provides a high-level summary of important configuration values
      • -
    • a new kudu tablet move command which moves a tablet replica from one tablet +
    • a new kudu tablet move command which moves a tablet replica from one tablet server to another
      • -
    • a new kudu local_replica data_size command which summarizes the space usage +
    • a new kudu local_replica data_size command which summarizes the space usage of a local tablet
    • all on-disk data is now checksummed by default, which provides error detection for improved confidence when running Kudu on unreliable hardware
      • @@ -165,6 +165,8 @@ repository.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/09/18/kudu-consistency-pt1.html ---------------------------------------------------------------------- diff --git a/2017/09/18/kudu-consistency-pt1.html b/2017/09/18/kudu-consistency-pt1.html index 3f40265..39f6c65 100644 --- a/2017/09/18/kudu-consistency-pt1.html +++ b/2017/09/18/kudu-consistency-pt1.html @@ -237,29 +237,29 @@ have increasing timestamps, depending on the user’s choices.

      Row mutations performed by a single client instance are guaranteed to have increasing timestamps thus reflecting their potential causal relationship. This property is always enforced. However there are two major “knobs” that are available to the user to make performance trade-offs, the -Read mode, and the External Consistency mode (see here +Read mode, and the External Consistency mode (see here for more information on how to use the relevant APIs).

      -

      The first and most important knob, the Read mode, pertains to what is the guaranteed recency of +

      The first and most important knob, the Read mode, pertains to what is the guaranteed recency of data resulting from scans. Since Kudu uses replication for availability and fault-tolerance, there are always multiple replicas of any data item. Not all replicas must be up-to-date so if the user cares about recency, e.g. if the user requires that any data read includes all previously written data from a single client instance then it must -choose the READ_AT_SNAPSHOT read mode. With this mode enabled the client is guaranteed to observe +choose the READ_AT_SNAPSHOT read mode. With this mode enabled the client is guaranteed to observe “READ YOUR OWN WRITES” semantics, i.e. scans from a client will always include all previous mutations performed by that client. Note that this property is local to a single client instance, not a global property.

      -

      The second “knob”, the External Consistency mode, defines the semantics of how reads and writes -are performed across multiple client instances. By default, External Consistency is set to - CLIENT_PROPAGATED, meaning it’s up to the user to coordinate a set of timestamp tokens with clients (even +

      The second “knob”, the External Consistency mode, defines the semantics of how reads and writes +are performed across multiple client instances. By default, External Consistency is set to + CLIENT_PROPAGATED, meaning it’s up to the user to coordinate a set of timestamp tokens with clients (even across different machines) if they are performing writes/reads that are somehow causally linked. If done correctly this enables STRICT SERIALIZABILITY[5], i.e. LINEARIZABILITY[6] and SERIALIZABILITY[7] at the same time, at the cost of having the user coordinate the timestamp tokens across clients (a survey of the meaning of these, and other definitions can be found here). -The alternative setting for External Consistency is to have it set to -COMMIT_WAIT (experimental), which guarantees the same properties through a different means, by +The alternative setting for External Consistency is to have it set to +COMMIT_WAIT (experimental), which guarantees the same properties through a different means, by implementing Google Spanner’s TrueTime. This comes at the cost of higher latency (depending on how tightly synchronized the system clocks of the various tablet servers are), but doesn’t require users to propagate timestamps programmatically.

      @@ -302,6 +302,8 @@ to enable the consistency semantics introduced in the previous section, includin

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/10/23/nosql-kudu-spanner-slides.html ---------------------------------------------------------------------- diff --git a/2017/10/23/nosql-kudu-spanner-slides.html b/2017/10/23/nosql-kudu-spanner-slides.html index 48e7a43..0deb771 100644 --- a/2017/10/23/nosql-kudu-spanner-slides.html +++ b/2017/10/23/nosql-kudu-spanner-slides.html @@ -183,6 +183,8 @@ below:

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/12/08/apache-kudu-1-6-0-released.html ---------------------------------------------------------------------- diff --git a/2017/12/08/apache-kudu-1-6-0-released.html b/2017/12/08/apache-kudu-1-6-0-released.html index e82f170..59f6977 100644 --- a/2017/12/08/apache-kudu-1-6-0-released.html +++ b/2017/12/08/apache-kudu-1-6-0-released.html @@ -184,6 +184,8 @@ Maven repository and are

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/03/23/apache-kudu-1-7-0-released.html ---------------------------------------------------------------------- diff --git a/2018/03/23/apache-kudu-1-7-0-released.html b/2018/03/23/apache-kudu-1-7-0-released.html index b495689..9017af5 100644 --- a/2018/03/23/apache-kudu-1-7-0-released.html +++ b/2018/03/23/apache-kudu-1-7-0-released.html @@ -196,6 +196,8 @@ Maven repository and are

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/07/10/instrumentation-in-kudu.html ---------------------------------------------------------------------- diff --git a/2018/07/10/instrumentation-in-kudu.html b/2018/07/10/instrumentation-in-kudu.html index f42eeba..f019b04 100644 --- a/2018/07/10/instrumentation-in-kudu.html +++ b/2018/07/10/instrumentation-in-kudu.html @@ -156,6 +156,8 @@ below. My talk spans the first 34 minutes.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/08/06/getting-started-with-kudu-an-oreilly-title.html ---------------------------------------------------------------------- diff --git a/2018/08/06/getting-started-with-kudu-an-oreilly-title.html b/2018/08/06/getting-started-with-kudu-an-oreilly-title.html index 95f74a1..8e5916f 100644 --- a/2018/08/06/getting-started-with-kudu-an-oreilly-title.html +++ b/2018/08/06/getting-started-with-kudu-an-oreilly-title.html @@ -132,9 +132,9 @@ challenge at that time. In that context, on October 11th 2012 Todd Lipcon perform Apache Kudu’s initial commit. The commit message was:

      -
      Code for writing cfiles seems to basically work
+
      Code for writing cfiles seems to basically work
 Need to write code for reading cfiles, still
-
      +

      And Kudu development was off and running. Around this same time Todd, on his internal Wiki page, started listing out the papers he was reading to develop @@ -170,7 +170,7 @@ of Kudu. Specifically you will learn:

      Looking forward, I am excited to see Kudu gain additional features and adoption and eventually the second revision of this title. In the meantime, if you have -feedback or questions, please reach out on the #getting-started-kudu channel of +feedback or questions, please reach out on the #getting-started-kudu channel of the Kudu Slack or if you prefer non-real-time communication, please use the user@ mailing list!

      @@ -183,6 +183,8 @@ communication, please use the user@ mailing list!

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/09/11/simplified-pipelines-with-kudu.html ---------------------------------------------------------------------- diff --git a/2018/09/11/simplified-pipelines-with-kudu.html b/2018/09/11/simplified-pipelines-with-kudu.html index d8c92c1..69c9985 100644 --- a/2018/09/11/simplified-pipelines-with-kudu.html +++ b/2018/09/11/simplified-pipelines-with-kudu.html @@ -166,6 +166,8 @@ the backend.

      Recent posts

      http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/09/26/index-skip-scan-optimization-in-kudu.html ---------------------------------------------------------------------- diff --git a/2018/09/26/index-skip-scan-optimization-in-kudu.html b/2018/09/26/index-skip-scan-optimization-in-kudu.html new file mode 100644 index 0000000..bf62385 --- /dev/null +++ b/2018/09/26/index-skip-scan-optimization-in-kudu.html @@ -0,0 +1,320 @@ + + + + + + + + + + Apache Kudu - Index Skip Scan Optimization in Kudu + + + + + + + + + + + + + + + + + +
      + + + +
      +
      +

      Apache Kudu Blog

      +
      +
      + +
      +
      +
      +
      +

      Index Skip Scan Optimization in Kudu

      +

      Posted 26 Sep 2018 by Anupama Gupta

      +
      +
      +

      This summer I got the opportunity to intern with the Apache Kudu team at Cloudera. +My project was to optimize the Kudu scan path by implementing a technique called +index skip scan (a.k.a. scan-to-seek, see section 4.1 in [1]). I wanted to share +my experience and the progress we’ve made so far on the approach.

      + + + +

      Let’s begin with discussing the current query flow in Kudu. +Consider the following table:

      + +
      CREATE TABLE metrics (
+    host STRING,
+    tstamp INT,
+    clusterid INT,
+    role STRING,
+    PRIMARY KEY (host, tstamp, clusterid)
+);
      + +

      png +Sample rows of table metrics (sorted by key columns).

      + +

      In this case, by default, Kudu internally builds a primary key index (implemented as a +B-tree) for the table metrics. +As shown in the table above, the index data is sorted by the composite of all key columns. +When the user query contains the first key column (host), Kudu uses the index (as the index data is +primarily sorted on the first key column).

      + +

      Now, what if the user query does not contain the first key column and instead only contains the tstamp column? +In the above case, the tstamp column values are sorted with respect to host, +but are not globally sorted, and as such, it’s non-trivial to use the index to filter rows. +Instead, a full tablet scan is done by default. Other databases may optimize such scans by building secondary indexes +(though it might be redundant to build one on one of the primary keys). However, this isn’t an option for Kudu, +given its lack of secondary index support.

      + +

      The question is, can Kudu do better than a full tablet scan here?

      + +

      The answer is yes! Let’s observe the column preceding the tstamp column. We will refer to it as the +“prefix column” and its specific value as the “prefix key”. In this example, host is the prefix column. +Note that the prefix keys are sorted in the index and that all rows of a given prefix key are also sorted by the +remaining key columns. Therefore, we can use the index to skip to the rows that have distinct prefix keys, +and also satisfy the predicate on the tstamp column. +For example, consider the query:

      + +
      SELECT clusterid FROM metrics WHERE tstamp = 100;
      + +

      png +Skip scan flow illustration. The rows in green are scanned and the rest are skipped.

      + +

      The tablet server can use the index to skip to the first row with a distinct prefix key (host = helium) that +matches the predicate (tstamp = 100) and then scan through the rows until the predicate no longer matches. At that +point we would know that no more rows with host = helium will satisfy the predicate, and we can skip to the next +prefix key. This holds true for all distinct keys of host. Hence, this method is popularly known as +skip scan optimization[2, 3].

      + +

      Performance

      + +

      This optimization can speed up queries significantly, depending on the cardinality (number of distinct values) of the +prefix column. The lower the prefix column cardinality, the better the skip scan performance. In fact, when the +prefix column cardinality is high, skip scan is not a viable approach. The performance graph (obtained using the example +schema and query pattern mentioned earlier) is shown below.

      + +

      Based on our experiments, on up to 10 million rows per tablet (as shown below), we found that the skip scan performance +begins to get worse with respect to the full tablet scan performance when the prefix column cardinality +exceeds sqrt(number_of_rows_in_tablet). +Therefore, in order to use skip scan performance benefits when possible and maintain a consistent performance in cases +of large prefix column cardinality, we have tentatively chosen to dynamically disable skip scan when the number of skips for +distinct prefix keys exceeds sqrt(number_of_rows_in_tablet). +It will be an interesting project to further explore sophisticated heuristics to decide when +to dynamically disable skip scan.

      + +

      png

      + +

      Conclusion

      + +

      Skip scan optimization in Kudu can lead to huge performance benefits that scale with the size of +data in Kudu tablets. This is a work-in-progress patch. +The implementation in the patch works only for equality predicates on the non-first primary key +columns. An important point to note is that although, in the above specific example, the number of prefix +columns is one (host), this approach is generalized to work with any number of prefix columns.

      + +

      This work also lays the groundwork to leverage the skip scan approach and optimize query processing time in the +following use cases:

      + +
        +
      • Range predicates
        • +
      • In-list predicates
        • +
      + +

      This was my first time working on an open source project. I thoroughly enjoyed working on this challenging problem, +right from understanding the scan path in Kudu to working on a full-fledged implementation of +the skip scan optimization. I am very grateful to the Kudu team for guiding and supporting me throughout the +internship period.

      + +

      References

      + +

      [1]: Gupta, Ashish, et al. “Mesa: +Geo-replicated, near real-time, scalable data warehousing.” Proceedings of the VLDB Endowment 7.12 (2014): 1259-1270.

      + +

      [2]: Index Skip Scanning - Oracle Database

      + +

      [3]: Skip Scan - SQLite

      + + +
      +
      + + +
      +
      +

      Recent posts

      + +
      +
      + +
      +
      +
      +

      + Copyright © 2016 The Apache Software Foundation. +

      +

      + Apache Kudu, Kudu, Apache, the Apache feather logo, and the Apache Kudu + project logo are either registered trademarks or trademarks of The + Apache Software Foundation in the United States and other countries. +

      +
      +
      + + + +
      +
      +
      +
      + + + + + + + + + http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/index.html ---------------------------------------------------------------------- diff --git a/blog/index.html b/blog/index.html index cefe6a4..f70bfea 100644 --- a/blog/index.html +++ b/blog/index.html @@ -117,6 +117,29 @@
      +

      Index Skip Scan Optimization in Kudu

      +

      Posted 26 Sep 2018 by Anupama Gupta

      +
      +
      + +

      This summer I got the opportunity to intern with the Apache Kudu team at Cloudera. +My project was to optimize the Kudu scan path by implementing a technique called +index skip scan (a.k.a. scan-to-seek, see section 4.1 in [1]). I wanted to share +my experience and the progress we’ve made so far on the approach.

      + + + +
      +
      + Read full post... +
      +
      + + + + +
      +

      Simplified Data Pipelines with Kudu

      Posted 11 Sep 2018 by Mac Noland

      @@ -219,31 +242,6 @@ optimizations, incremental improvements, and bug fixes.

      - -
      -
      -

      Apache Kudu 1.6.0 released

      -

      Posted 08 Dec 2017 by Mike Percy

      -
      -
      - -

      The Apache Kudu team is happy to announce the release of Kudu 1.6.0!

      - -

      Apache Kudu 1.6.0 is a minor release that offers new features, performance -optimizations, incremental improvements, and bug fixes.

      - -

      Release highlights:

      - - - -
      -
      - Read full post... -
      -
      - - -