Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0FB30200BAE for ; Fri, 28 Oct 2016 18:57:57 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0E4C2160AE4; Fri, 28 Oct 2016 16:57:57 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D3395160ACA for ; Fri, 28 Oct 2016 18:57:54 +0200 (CEST) Received: (qmail 14234 invoked by uid 500); 28 Oct 2016 16:57:54 -0000 Mailing-List: contact commits-help@fluo.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@fluo.incubator.apache.org Delivered-To: mailing list commits@fluo.incubator.apache.org Received: (qmail 14224 invoked by uid 99); 28 Oct 2016 16:57:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Oct 2016 16:57:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 84008C13A4 for ; Fri, 28 Oct 2016 16:57:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 6mCUlqjiHEUM for ; Fri, 28 Oct 2016 16:57:43 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id E9E555FC59 for ; Fri, 28 Oct 2016 16:57:39 +0000 (UTC) Received: (qmail 13843 invoked by uid 99); 28 Oct 2016 16:57:38 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Oct 2016 16:57:38 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 93050F16C4; Fri, 28 Oct 2016 16:57:38 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: mwalch@apache.org To: commits@fluo.incubator.apache.org Date: Fri, 28 Oct 2016 16:57:45 -0000 Message-Id: <5221f630192340488d65af4c93faa96e@git.apache.org> In-Reply-To: <4ae001eae02741b8b0719d89fa6914d0@git.apache.org> References: <4ae001eae02741b8b0719d89fa6914d0@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [8/8] incubator-fluo-website git commit: Jekyll build from gh-pages:887634d archived-at: Fri, 28 Oct 2016 16:57:57 -0000 Jekyll build from gh-pages:887634d Merge branch 'recipes-1.0.0-incubating' into gh-pages Project: http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/commit/f6125fa2 Tree: http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/tree/f6125fa2 Diff: http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/diff/f6125fa2 Branch: refs/heads/asf-site Commit: f6125fa246d864463e7e6ab2c3fb2b77f26bd21b Parents: 90818a6 Author: Mike Walch Authored: Fri Oct 28 12:55:19 2016 -0400 Committer: Mike Walch Committed: Fri Oct 28 12:55:19 2016 -0400 ---------------------------------------------------------------------- README.md | 2 +- api/archive/index.html | 4 +- api/index.html | 2 +- docs/fluo-recipes/1.0.0-beta-1/cfm/index.html | 2 +- .../1.0.0-beta-1/export-queue/index.html | 2 +- docs/fluo-recipes/1.0.0-beta-1/index.html | 2 +- .../1.0.0-beta-1/recording-tx/index.html | 2 +- .../1.0.0-beta-1/serialization/index.html | 2 +- .../1.0.0-beta-1/table-optimization/index.html | 2 +- .../1.0.0-beta-1/transient/index.html | 2 +- .../1.0.0-beta-2/accumulo-export/index.html | 2 + docs/fluo-recipes/1.0.0-beta-2/cfm/index.html | 2 + .../1.0.0-beta-2/export-queue/index.html | 2 + docs/fluo-recipes/1.0.0-beta-2/index.html | 2 + .../1.0.0-beta-2/recording-tx/index.html | 2 + .../1.0.0-beta-2/row-hasher/index.html | 2 + .../1.0.0-beta-2/serialization/index.html | 2 + .../1.0.0-beta-2/table-optimization/index.html | 2 + .../1.0.0-beta-2/testing/index.html | 2 + .../1.0.0-beta-2/transient/index.html | 2 + .../accumulo-export-queue/index.html | 216 ++++++++++ .../1.0.0-incubating/cfm/index.html | 364 +++++++++++++++++ .../1.0.0-incubating/export-queue/index.html | 409 +++++++++++++++++++ docs/fluo-recipes/1.0.0-incubating/index.html | 202 +++++++++ .../1.0.0-incubating/recording-tx/index.html | 187 +++++++++ .../1.0.0-incubating/row-hasher/index.html | 235 +++++++++++ .../1.0.0-incubating/serialization/index.html | 188 +++++++++ .../table-optimization/index.html | 177 ++++++++ .../1.0.0-incubating/testing/index.html | 129 ++++++ .../1.0.0-incubating/transient/index.html | 196 +++++++++ docs/index.html | 2 +- feed.xml | 74 +++- index.html | 12 +- news/index.html | 5 + .../fluo-recipes-1.0.0-incubating/index.html | 163 ++++++++ release/index.html | 7 +- 36 files changed, 2571 insertions(+), 37 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index ecbb3b8..b5fbaf1 100644 --- a/README.md +++ b/README.md @@ -54,7 +54,7 @@ Steps to update website for new Fluo Recipes release: ```bash cd fluo-website - mkdir -p docs/1.0.0-beta-1 + mkdir -p docs/fluo-recipes/1.0.0-beta-1 ./_scripts/convert-recipes-docs.py /path/to/fluo-recipes/docs/ /path/to/fluo-website/docs/fluo-recipes/1.0.0-beta-1/ ``` http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/api/archive/index.html ---------------------------------------------------------------------- diff --git a/api/archive/index.html b/api/archive/index.html index 1303c21..df6988f 100644 --- a/api/archive/index.html +++ b/api/archive/index.html @@ -92,8 +92,8 @@

Fluo Recipes API documentation (before Apache)

http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/api/index.html ---------------------------------------------------------------------- diff --git a/api/index.html b/api/index.html index 6da3ac9..50b9792 100644 --- a/api/index.html +++ b/api/index.html @@ -90,7 +90,7 @@

Apache Fluo Recipes API

    -
  • The first release of Apache Fluo Recipes has not been made yet.
    • +
  • 1.0.0-incubating: core, accumulo, kryo, spark, test - October 28, 2016

API for releases before joining Apache have been archived.

http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-1/cfm/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-1/cfm/index.html b/docs/fluo-recipes/1.0.0-beta-1/cfm/index.html index 7f64b18..da338bc 100644 --- a/docs/fluo-recipes/1.0.0-beta-1/cfm/index.html +++ b/docs/fluo-recipes/1.0.0-beta-1/cfm/index.html @@ -75,7 +75,7 @@
- + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-1/export-queue/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-1/export-queue/index.html b/docs/fluo-recipes/1.0.0-beta-1/export-queue/index.html index 944a2c9..08ad02b 100644 --- a/docs/fluo-recipes/1.0.0-beta-1/export-queue/index.html +++ b/docs/fluo-recipes/1.0.0-beta-1/export-queue/index.html @@ -75,7 +75,7 @@
- + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-1/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-1/index.html b/docs/fluo-recipes/1.0.0-beta-1/index.html index b7089ce..aaa9203 100644 --- a/docs/fluo-recipes/1.0.0-beta-1/index.html +++ b/docs/fluo-recipes/1.0.0-beta-1/index.html @@ -75,7 +75,7 @@
- + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-1/recording-tx/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-1/recording-tx/index.html b/docs/fluo-recipes/1.0.0-beta-1/recording-tx/index.html index d11b407..86c29b0 100644 --- a/docs/fluo-recipes/1.0.0-beta-1/recording-tx/index.html +++ b/docs/fluo-recipes/1.0.0-beta-1/recording-tx/index.html @@ -75,7 +75,7 @@
- + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-1/serialization/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-1/serialization/index.html b/docs/fluo-recipes/1.0.0-beta-1/serialization/index.html index faaf44c..d9063be 100644 --- a/docs/fluo-recipes/1.0.0-beta-1/serialization/index.html +++ b/docs/fluo-recipes/1.0.0-beta-1/serialization/index.html @@ -75,7 +75,7 @@
- + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-1/table-optimization/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-1/table-optimization/index.html b/docs/fluo-recipes/1.0.0-beta-1/table-optimization/index.html index 505b8e9..8ab0f1e 100644 --- a/docs/fluo-recipes/1.0.0-beta-1/table-optimization/index.html +++ b/docs/fluo-recipes/1.0.0-beta-1/table-optimization/index.html @@ -75,7 +75,7 @@
- + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-1/transient/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-1/transient/index.html b/docs/fluo-recipes/1.0.0-beta-1/transient/index.html index c8bddf0..e3b26c8 100644 --- a/docs/fluo-recipes/1.0.0-beta-1/transient/index.html +++ b/docs/fluo-recipes/1.0.0-beta-1/transient/index.html @@ -75,7 +75,7 @@
- + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/accumulo-export/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/accumulo-export/index.html b/docs/fluo-recipes/1.0.0-beta-2/accumulo-export/index.html index 132df8b..b651a66 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/accumulo-export/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/accumulo-export/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/cfm/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/cfm/index.html b/docs/fluo-recipes/1.0.0-beta-2/cfm/index.html index 2af6ea0..e4a4ad2 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/cfm/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/cfm/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/export-queue/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/export-queue/index.html b/docs/fluo-recipes/1.0.0-beta-2/export-queue/index.html index 8246e91..49fdfde 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/export-queue/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/export-queue/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/index.html b/docs/fluo-recipes/1.0.0-beta-2/index.html index dcd725c..fd12b2a 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/recording-tx/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/recording-tx/index.html b/docs/fluo-recipes/1.0.0-beta-2/recording-tx/index.html index 897fe3b..c9eda7d 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/recording-tx/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/recording-tx/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/row-hasher/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/row-hasher/index.html b/docs/fluo-recipes/1.0.0-beta-2/row-hasher/index.html index be3d54e..cc38295 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/row-hasher/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/row-hasher/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/serialization/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/serialization/index.html b/docs/fluo-recipes/1.0.0-beta-2/serialization/index.html index e910e7b..9bf96f1 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/serialization/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/serialization/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/table-optimization/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/table-optimization/index.html b/docs/fluo-recipes/1.0.0-beta-2/table-optimization/index.html index 5e262b5..4c87933 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/table-optimization/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/table-optimization/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/testing/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/testing/index.html b/docs/fluo-recipes/1.0.0-beta-2/testing/index.html index a367e3f..3ed0923 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/testing/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/testing/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-beta-2/transient/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-beta-2/transient/index.html b/docs/fluo-recipes/1.0.0-beta-2/transient/index.html index bfdf350..eda4c09 100644 --- a/docs/fluo-recipes/1.0.0-beta-2/transient/index.html +++ b/docs/fluo-recipes/1.0.0-beta-2/transient/index.html @@ -75,6 +75,8 @@
+ + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-incubating/accumulo-export-queue/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-incubating/accumulo-export-queue/index.html b/docs/fluo-recipes/1.0.0-incubating/accumulo-export-queue/index.html new file mode 100644 index 0000000..8ad505c --- /dev/null +++ b/docs/fluo-recipes/1.0.0-incubating/accumulo-export-queue/index.html @@ -0,0 +1,216 @@ + + + + + + + + + + + + + Accumulo Export Queue Specialization | Apache Fluo + + + + +
+
+
+
+ + + + + +
+

Accumulo Export Queue Specialization

+
+ +
+

Background

+ +

The Export Queue Recipe provides a generic foundation for building export mechanism to any +external data store. The AccumuloExporter provides an implementation of this recipe for +Accumulo. The AccumuloExporter is located the fluo-recipes-accumulo module and provides the +following functionality:

+ +
    +
  • Safely batches writes to Accumulo made by multiple transactions exporting data.
    • +
  • Stores Accumulo connection information in Fluo configuration, making it accessible by Export +Observers running on other nodes.
    • +
  • Provides utility code that make it easier and shorter to code common Accumulo export patterns.
    • +
+ +

Example Use

+ +

Exporting to Accumulo is easy. Follow the steps below:

+ +
    +
  1. +

    Implement a class that extends AccumuloExporter. This class will process exported objects that +are placed on your export queue. For example, the SimpleExporter class below processes String +key/value exports and generates mutations for Accumulo.

    + +

    ```java + public class SimpleExporter extends AccumuloExporter<String, String> {

    + +

    @Override + protected void translate(SequencedExport<String, String> export, Consumer consumer) { + Mutation m = new Mutation(export.getKey()); + m.put("cf", "cq", export.getSequence(), export.getValue()); + consumer.accept(m); + } + } + ```

    +
    2. +
  2. +

    With a SimpleExporter created, configure an ExportQueue to use SimpleExporter and +give it information on how to connect to Accumulo.

    + +

    ```java

    + +

    FluoConfiguration fluoConfig = …;

    + +

    // Set accumulo configuration + String instance = // Name of accumulo instance exporting to + String zookeepers = // Zookeepers used by Accumulo instance exporting to + String user = // Accumulo username, user that can write to exportTable + String password = // Accumulo user password + String exportTable = // Name of table to export to

    + +

    // Create config for export table. + AccumuloExporter.Configuration exportTableCfg = + new AccumuloExporter.Configuration(instance, zookeepers, user, password, exportTable);

    + +

    // Create config for export queue. + ExportQueue.Options eqOpts = new ExportQueue.Options(EXPORT_QUEUE_ID, SimpleExporter.class, + String.class, String.class, numMapBuckets).setExporterConfiguration(exportTableCfg);

    + +

    // Configure export queue. This will modify fluoConfig. + ExportQueue.configure(fluoConfig, qeOpts);

    + +

    // Initialize Fluo using fluoConfig + ```

    +
    3. +
  3. +

    Export queues can be retrieved in Fluo observers and objects can be added to them:

    + + 
    public class MyObserver extends AbstractObserver {
+
+  ExportQueue<String, String> exportQ;
+
+  @Override
+  public void init(Context context) throws Exception {
+    exportQ = ExportQueue.getInstance(EXPORT_QUEUE_ID, context.getAppConfiguration());
+  }
+
+  @Override
+  public void process(TransactionBase tx, Bytes row, Column col) {
+
+    // Read some data and do some work
+
+    // Add results to export queue
+    String key =    // key that identifies export
+    String value =  // object to export
+    export.add(tx, key, value);
+  }
+
    +
    +
    4. +
+ +

Other use cases

+ +

AccumuloReplicator is a specialized AccumuloExporter that replicates a Fluo table to Accumulo.

+ + +
+ +
+ +
+
+
+
+
+
+ Apache + Copyright © 2016 The Apache Software Foundation. Licensed under the Apache License, Version 2.0 +
+
+
+
+ + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/f6125fa2/docs/fluo-recipes/1.0.0-incubating/cfm/index.html ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/1.0.0-incubating/cfm/index.html b/docs/fluo-recipes/1.0.0-incubating/cfm/index.html new file mode 100644 index 0000000..baccd0b --- /dev/null +++ b/docs/fluo-recipes/1.0.0-incubating/cfm/index.html @@ -0,0 +1,364 @@ + + + + + + + + + + + + + Collision Free Map Recipe | Apache Fluo + + + + +
+
+
+
+ + + + + +
+

Collision Free Map Recipe

+
+ +
+

Background

+ +

When many transactions are trying to modify the same keys, collisions will occur. +These collisions will cause the transactions to fail and throughput to nose +dive. For example consider the phrasecount example. In this example many +transactions are processing documents as input. Each transaction counts the +phrases in its document and then tries to update global phrase counts. With +each transaction attempting to update many phrase counts, the probability of +two transactions colliding is very high.

+ +

Solution

+ +

This recipe provides a reusable solution for the problem of many transactions +updating many keys while avoiding collisions. As an added bonus, this recipe +also organizes updates into batches for efficiency in order to improve +throughput.

+ +

The central idea behind this recipe is that updates to a key are queued up to +be processed by another transaction triggered by weak notifications. In the +phrase count example transactions processing documents would queue updates, +but would not actually update the counts. Below is an example of how +transactions would compute phrasecounts using this recipe.

+ +
    +
  • TX1 queues +1 update for phrase we want lambdas now
    • +
  • TX2 queues +1 update for phrase we want lambdas now
    • +
  • TX3 reads the updates and current value for the phrase we want lambdas now. There is no current value and the updates sum to 2, so a new value of 2 is written.
    • +
  • TX4 queues +2 update for phrase we want lambdas now
    • +
  • TX5 queues -1 update for phrase we want lambdas now
    • +
  • TX6 reads the updates and current value for the phrase we want lambdas now. The current value is 2 and the updates sum to 1, so a new value of 3 is written.
    • +
+ +

Transactions processing updates have the ability to make additional updates. +For example in addition to updating the current value for a phrase, the new +value could also be placed on an export queue to update an external database.

+ +

Buckets

+ +

A simple implementation of this recipe would be to have an update queue for +each key. However the implementation does something slightly more complex. +Each update queue is in a bucket and transactions that process updates, process +all of the updates in a bucket. This allows more efficient processing of +updates for the following reasons :

+ +
    +
  • When updates are queued, notifications are made per bucket(instead of per a key).
    • +
  • The transaction doing the update can scan the entire bucket reading updates, this avoids a seek for each key being updated.
    • +
  • Also the transaction can request a batch lookup to get the current value of all the keys being updated.
    • +
  • Any additional actions taken on update (like adding something to an export queue) can also be batched.
    • +
  • Data is organized to make reading exiting values for keys in a bucket more efficient.
    • +
+ +

Which bucket a key goes to is decided using hash and modulus so that multiple +updates for the same key always go to the same bucket.

+ +

The initial number of tablets to create when applying table optimizations can be +controlled by setting the buckets per tablet option when configuring a Collision +Free Map. For example if you have 20 tablet servers and 1000 buckets and want +2 tablets per tserver initially then set buckets per tablet to 1000/(2*20)=25.

+ +

Example Use

+ +

The following code snippets show how to setup and use this recipe for +wordcount. The first step in using this recipe is to configure it before +initializing Fluo. When initializing an ID will need to be provided. This ID +is used in two ways. First, the ID is used as a row prefix in the table. +Therefore nothing else should use that row range in the table. Second, the ID +is used in generating configuration keys associated with the instance of the +Collision Free Map.

+ +

The following snippet shows how to setup a collision free map.

+ +
  FluoConfiguration fluoConfig = ...;
+
+  int numBuckets = 119;
+
+  WordCountMap.configure(fluoConfig, 119);
+
+  //initialize Fluo using fluoConfig
+
+
+
+ +

Assume the following observer is triggered when a documents contents are +updated. It examines new and old document content and determines changes in +word counts. These changes are pushed to a collision free map.

+ +
public class DocumentObserver extends TypedObserver {
+
+  CollisionFreeMap<String, Long> wcm;
+
+  @Override
+  public void init(Context context) throws Exception {
+    wcm = CollisionFreeMap.getInstance(WordCountMap.ID, context.getAppConfiguration());
+  }
+
+  @Override
+  public ObservedColumn getObservedColumn() {
+    return new ObservedColumn(new Column("content", "new"), NotificationType.STRONG);
+  }
+
+  @Override
+  public void process(TypedTransactionBase tx, Bytes row, Column col) {
+    String newContent = tx.get().row(row).col(col).toString();
+    String currentContent = tx.get().row(row).fam("content").qual("current").toString("");
+
+    Map<String, Long> newWordCounts = getWordCounts(newContent);
+    Map<String, Long> currentWordCounts = getWordCounts(currentContent);
+
+    //determine changes in word counts between old and new document content
+    Map<String, Long> changes = calculateChanges(newWordCounts, currentWordCounts);    
+
+    //queue updates to word counts for processing by other transactions
+    wcm.update(tx, changes);
+
+    //update the current content and delete the new content
+    tx.mutate().row(row).fam("content").qual("current").set(newContent);
+    tx.mutate().row(row).col(col).delete();
+  }
+
+  private static Map<String, Long> getWordCounts(String doc) {
+   //TODO extract words from doc
+  }
+
+  private static Map<String, Long> calculateChanges(Map<String, Long> newCounts,
+      Map<String, Long> currCounts) {
+    Map<String, Long> changes = new HashMap<>();
+
+    // guava Maps class
+    MapDifference<String, Long> diffs = Maps.difference(currCounts, newCounts);
+
+    // compute the diffs for words that changed
+    changes.putAll(Maps.transformValues(diffs.entriesDiffering(),
+        vDiff -> vDiff.rightValue() - vDiff.leftValue()));
+
+    // add all new words
+    changes.putAll(diffs.entriesOnlyOnRight());
+
+    // subtract all words no longer present
+    changes.putAll(Maps.transformValues(diffs.entriesOnlyOnLeft(), l -> l * -1));
+
+    return changes;
+  }
+
+}
+
+
+ +

Each collision free map has two extension points, a combiner and an update +observer. These two extension points are defined below as WordCountCombiner +and WordCountObserver. The collision free map configures a Fluo observer that +will process queued updates. When processing these queued updates the two +extension points are called. In this example WordCountCombiner is called to +combine updates that were queued by DocumentObserver. The collision free map +will process a batch of keys, calling the combiner for each key. When finished +processing a batch, it will call the update observer WordCountObserver.

+ +

An update observer can do additional processing when a batch of key values are +updated. In WordCountObserver, updates are queued for export to an external +database. The export is given the new and old value allowing it to delete the +old value if needed.

+ +
/**
+ * This class exists to provide a single place to put all code related to the
+ * word count map.
+ */
+public class WordCountMap {
+
+  public static final String ID = "wc";
+
+  /**
+   * A helper method for configuring the word count map.
+   *
+   * @param numTablets the desired number of tablets to create when applying table optimizations
+   */
+  public static void configure(FluoConfiguration fluoConfig, int numBuckets, int numTablets) {
+    Options cfmOpts =
+      new Options(ID, WordCountCombiner.class,  WordCountObserver.class, String.class, Long.class, numBuckets)
+        .setBucketsPerTablet(numBuckets/numTablets);
+    CollisionFreeMap.configure(fluoConfig, cfmOpts);
+  }
+
+  public static class WordCountCombiner implements Combiner<String, Long> {
+    @Override
+    public Optional<Long> combine(String key, Iterator<Long> updates) {
+      long sum = 0L;
+
+      while (updates.hasNext()) {
+        sum += updates.next();
+      }
+
+      if (sum == 0) {
+        //returning absent will cause the collision free map to delte the current key
+        return Optional.absent();
+      } else {
+        return Optional.of(sum);
+      }
+    }
+  }
+
+  public static class WordCountObserver extends UpdateObserver<String, Long> {
+
+    private ExportQueue<String, MyDatabaseExport> exportQ;
+
+    @Override
+    public void init(String mapId, Context observerContext) throws Exception {
+      exportQ = ExportQueue.getInstance(MyExportQ.ID, observerContext.getAppConfiguration());
+    }
+
+    @Override
+    public void updatingValues(TransactionBase tx, Iterator<Update<String, Long>> updates) {
+      while (updates.hasNext()) {
+        Update<String, Long> update = updates.next();
+
+        String word = update.getKey();
+        Optional<Long> oldVal = update.getOldValue();
+        Optional<Long> newVal = update.getNewValue();
+
+        //queue an export to let an external database know the word count has changed
+        exportQ.add(word, new MyDatabaseExport(oldVal, newVal));
+      }
+    }
+  }
+}
+
+
+ +

Guarantees

+ +

This recipe makes two important guarantees about updates for a key when it +calls updatingValues() on an UpdateObserver.

+ +
    +
  • The new value reported for an update will be derived from combining all +updates that were committed before the transaction thats processing updates +started. The implementation may have to make multiple passes over queued +updates to achieve this. In the situation where TX1 queues a +1 and later +TX2 queues a -1 for the same key, there is no need to worry about only seeing +the -1 processed. A transaction that started processing updates after TX2 +committed would process both.
    • +
  • The old value will always be what was reported as the new value in the +previous transaction that called updatingValues().
    • +
+ + +
+ +
+ +
+
+
+
+
+
+ Apache + Copyright © 2016 The Apache Software Foundation. Licensed under the Apache License, Version 2.0 +
+
+
+
+ + + + + + + + +