Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 40352200D54 for ; Fri, 8 Dec 2017 23:48:44 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 3EA4D160C0D; Fri, 8 Dec 2017 22:48:44 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 88EAC160BFD for ; Fri, 8 Dec 2017 23:48:42 +0100 (CET) Received: (qmail 75141 invoked by uid 500); 8 Dec 2017 22:48:41 -0000 Mailing-List: contact commits-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list commits@accumulo.apache.org Received: (qmail 75132 invoked by uid 99); 8 Dec 2017 22:48:41 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Dec 2017 22:48:41 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id EBFE38581E; Fri, 8 Dec 2017 22:48:39 +0000 (UTC) Date: Fri, 08 Dec 2017 22:48:39 +0000 To: "commits@accumulo.apache.org" Subject: [accumulo-website] branch asf-site updated: Jekyll build from master:52f812c MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <151277331915.18870.16266357569834855765@gitbox.apache.org> From: kturner@apache.org X-Git-Host: gitbox.apache.org X-Git-Repo: accumulo-website X-Git-Refname: refs/heads/asf-site X-Git-Reftype: branch X-Git-Oldrev: 9f99c133144d19f087ace34bf76bdd240ee3f96a X-Git-Newrev: 5ce4558c6d61acd9ac7c3fc7d8524b1fc9973a22 X-Git-Rev: 5ce4558c6d61acd9ac7c3fc7d8524b1fc9973a22 X-Git-NotificationType: ref_changed_plus_diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated archived-at: Fri, 08 Dec 2017 22:48:44 -0000 This is an automated email from the ASF dual-hosted git repository. kturner pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/accumulo-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 5ce4558 Jekyll build from master:52f812c 5ce4558 is described below commit 5ce4558c6d61acd9ac7c3fc7d8524b1fc9973a22 Author: Keith Turner AuthorDate: Fri Dec 8 17:47:09 2017 -0500 Jekyll build from master:52f812c Added conditional writer to tour --- feed.xml | 4 +- tour/authorizations-code/index.html | 4 +- tour/authorizations/index.html | 4 +- tour/basic-read-write/index.html | 4 +- tour/batch-scanner-code/index.html | 8 +- tour/batch-scanner/index.html | 4 +- .../index.html | 73 +++++------ tour/conditional-writer/index.html | 136 ++++++++++++++++++++- tour/data-model-code/index.html | 4 +- tour/data-model/index.html | 4 +- tour/getting-started/index.html | 4 +- tour/index.html | 6 + tour/ranges-splits/index.html | 4 +- tour/using-iterators/index.html | 4 +- 14 files changed, 198 insertions(+), 65 deletions(-) diff --git a/feed.xml b/feed.xml index fb6e542..e95efd2 100644 --- a/feed.xml +++ b/feed.xml @@ -6,8 +6,8 @@ https://accumulo.apache.org/ - Fri, 08 Dec 2017 17:02:42 -0500 - Fri, 08 Dec 2017 17:02:42 -0500 + Fri, 08 Dec 2017 17:46:55 -0500 + Fri, 08 Dec 2017 17:46:55 -0500 Jekyll v3.5.2 diff --git a/tour/authorizations-code/index.html b/tour/authorizations-code/index.html index b119a72..9e599dd 100644 --- a/tour/authorizations-code/index.html +++ b/tour/authorizations-code/index.html @@ -153,7 +153,7 @@

Accumulo Tour: Authorizations Code

-

Tour page 6 of 9

+

Tour page 6 of 11

Below is a solution for the exercise.

@@ -241,7 +241,7 @@ if (e.keyCode == '39') { window.location = '/tour/ranges-splits/'; } < - 6 / 9 + 6 / 11 > diff --git a/tour/authorizations/index.html b/tour/authorizations/index.html index 82b6734..f8f3fac 100644 --- a/tour/authorizations/index.html +++ b/tour/authorizations/index.html @@ -153,7 +153,7 @@

Accumulo Tour: Authorizations

-

Tour page 5 of 9

+

Tour page 5 of 11

Authorizations are a set of Strings that enable a user to read protected data. Users are granted authorizations @@ -229,7 +229,7 @@ if (e.keyCode == '39') { window.location = '/tour/authorizations-code/'; } < - 5 / 9 + 5 / 11 > diff --git a/tour/basic-read-write/index.html b/tour/basic-read-write/index.html index ad18351..5c29410 100644 --- a/tour/basic-read-write/index.html +++ b/tour/basic-read-write/index.html @@ -153,7 +153,7 @@

Accumulo Tour: Writing and Reading

-

Tour page 2 of 9

+

Tour page 2 of 11

Accumulo is a big key/value store. Writing data to Accumulo is flexible and fast. Like any database, Accumulo stores @@ -227,7 +227,7 @@ if (e.keyCode == '39') { window.location = '/tour/data-model/'; } < - 2 / 9 + 2 / 11 > diff --git a/tour/batch-scanner-code/index.html b/tour/batch-scanner-code/index.html index 11b0680..6e8559d 100644 --- a/tour/batch-scanner-code/index.html +++ b/tour/batch-scanner-code/index.html @@ -153,7 +153,7 @@

Accumulo Tour: Batch Scanner Code

-

Tour page 9 of 9

+

Tour page 9 of 11

Below is a solution to the exercise.

@@ -213,6 +213,8 @@ if (e.keyCode == '37') { window.location = '/tour/batch-scanner/'; } +if (e.keyCode == '39') { window.location = '/tour/conditional-writer/'; } + }; @@ -222,7 +224,9 @@ if (e.keyCode == '37') { window.location = '/tour/batch-scanner/'; } < - 9 / 9 + 9 / 11 + + >
diff --git a/tour/batch-scanner/index.html b/tour/batch-scanner/index.html index d52fb8d..adb857f 100644 --- a/tour/batch-scanner/index.html +++ b/tour/batch-scanner/index.html @@ -153,7 +153,7 @@

Accumulo Tour: Batch Scanner

-

Tour page 8 of 9

+

Tour page 8 of 11

Running on a single thread, a Scanner will retrieve a single Range of data and return Keys in sorted order. A BatchScanner @@ -219,7 +219,7 @@ if (e.keyCode == '39') { window.location = '/tour/batch-scanner-code/'; } < - 8 / 9 + 8 / 11 > diff --git a/tour/ranges-splits/index.html b/tour/conditional-writer-code/index.html similarity index 63% copy from tour/ranges-splits/index.html copy to tour/conditional-writer-code/index.html index a0f74d4..a1693e7 100644 --- a/tour/ranges-splits/index.html +++ b/tour/conditional-writer-code/index.html @@ -25,7 +25,7 @@ -Ranges and Splits +Conditional Writer Code @@ -146,75 +146,68 @@

-

Ranges and Splits

+

Conditional Writer Code

-

Accumulo Tour: Ranges and Splits

-

Tour page 7 of 9

+

Accumulo Tour: Conditional Writer Code

+

Tour page 11 of 11

-

A Range is a specified group of Keys. There are many different ways to create a Range. Here are a few examples:

+

Below is a solution to the excercise.

+ +
  static boolean setAddress(Connector conn, String id, String expectedAddr, String try (ConditionalWriter writer = conn.createConditionalWriter("GothamPD", new ConditionalWriterConfig())) {
+      Condition condition = new Condition("location", "home");
+      if(expectedAddr != null) {
+        condition.setValue(expectedAddr);
+      }
+      ConditionalMutation mutation = new ConditionalMutation(id, condition);
+      mutation.put("location", "home", newAddr);
+      return writer.write(mutation).getStatus() == Status.ACCEPTED;
+    } catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
 
-
Range r1 = new Range(startKey, endKey);  // Creates a range from startKey inclusive to endKey inclusive.
-Range r2 = new Range(row);               // Creates a range that covers an entire row.
-Range r3 = new Range(startRow, endRow);  // Creates a range from startRow inclusive to endRow inclusive.
 
-

A Scanner by default will scan all Keys in a table but this can be inefficient. It is a good practice to set a range on a Scanner.

- -
scanner.setRange(new Range("id0000", "id0010"));  // returns rows from id0000 to id0010
+

The following output shows running the example with a conditional writer. +Threads retry when conditional mutations are rejected. The final address has +all three modifications.

+ +
Thread  37 attempting change '  1007 Mountain Drive, Gotham, New York  ' -> '  1007 Mountain Dr, Gotham, New York  '
+Thread  38 attempting change '  1007 Mountain Drive, Gotham, New York  ' -> '1007 Mountain Drive, Gotham, New York'
+Thread  39 attempting change '  1007 Mountain Drive, Gotham, New York  ' -> '  1007 Mountain Drive, Gotham, NY  '
+Thread  38 attempting change '  1007 Mountain Dr, Gotham, New York  ' -> '1007 Mountain Dr, Gotham, New York'
+Thread  39 attempting change '  1007 Mountain Dr, Gotham, New York  ' -> '  1007 Mountain Dr, Gotham, NY  '
+Thread  39 attempting change '1007 Mountain Dr, Gotham, New York' -> '1007 Mountain Dr, Gotham, NY'
+Final address : '1007 Mountain Dr, Gotham, NY'
 
-

As your data grows larger, Accumulo will split tables into smaller pieces called Tablets. Tablets can then be distributed across multiple Tablet Servers.
-By default, a table will get split into Tablets on row boundaries, guaranteeing an entire row to be on one Tablet Server. We have the ability to -tell Accumulo where to split tables by setting split points. This is done using addSplits in the TableOperations API. The image below -demonstrates how Accumulo splits data.

- -

data distribution

- -

Take a minute to learn these Accumulo terms:

-
    -
  • Tablet - A partition of a table.
  • -
  • Split - A point where tables are partitioned into separate tablets.
  • -
  • Flush - Action taken when data is written from memory to disk.
  • -
  • Compact - Action taken when files on disk are consolidated.
  • -
  • Iterator - A server side mechanism that can filter and modify Key/Value pairs.
  • -
- -

Knowing these terms are critical when working closely with Accumulo. Iterators are especially unique and powerful. More on them later.

- -

When working with large amounts of data across many Tablet Servers, a simple Scanner might not do the trick. Next lesson we learn about the power of -the multi-threaded BatchScanner!

- -

- < + < - 7 / 9 - - > + 11 / 11

diff --git a/tour/conditional-writer/index.html b/tour/conditional-writer/index.html index 7b53dcb..a464a35 100644 --- a/tour/conditional-writer/index.html +++ b/tour/conditional-writer/index.html @@ -153,26 +153,156 @@

Accumulo Tour: Conditional Writer

-

Tour page of 9

+

Tour page 10 of 11

- +

Suppose the Gotham PD is storing home addresses for persons of interest in +Accumulo. We want to correctly handle the case of multiple users editing the +same address at the same time. The following sequence of events shows an example +of how this can go wrong.

+ +
    +
  1. User 0 sets the key id0001:location:home to 1007 Mountain Drive, Gotham, New York
  2. +
  3. User 1 reads id0001:location:home
  4. +
  5. User 2 reads id0001:location:home
  6. +
  7. User 1 replaces Drive with Dr
  8. +
  9. User 2 replaces New York with NY
  10. +
  11. User 1 sets key id0001:location:home to 1007 Mountain Dr, Gotham, New York
  12. +
  13. User 2 sets key id0001:location:home to 1007 Mountain Drive, Gotham, NY
  14. +
+ +

In this situation the changes made by User 1 are lost, ending up with 1007 +Mountain Drive, Gotham, NY instead of 1007 Mountain Dr, Gotham, NY. To +correctly handle this, Accumulo offers the ConditionalWriter. The +ConditionalWriter atomically checks conditions on a row and only applies a +mutation when all conditions are satisfied.

+ +

Exercise

+ +

The following code simulates the concurrency in the situation above. The code +starts multiple threads, with each thread doing the following.

+ +
    +
  1. Read key’s value into memory using a scanner
  2. +
  3. Modify the copy in memory.
  4. +
  5. Write out the modified value from memory using a batch writer.
  6. +
  7. If write was unsuccessful, then goto step 1.
  8. +
+ +

This process can result in threads overwriting each other changes. The problem +is the batch writer always makes the update, even when the value has +changed since it was read.

+ +
  static String getAddress(Connector conn, String id) {
+    // The IsolatedScanner ensures partial changes to a row are not seen
+    try (Scanner scanner = new IsolatedScanner(conn.createScanner("GothamPD", Authorizations.EMPTY< [...]
+      scanner.setRange(Range.exact(id, "location", "home"));
+      for (Entry<Key,Value> entry : scanner) {
+        return entry.getValue().toString();
+      }
+      return null;
+    } catch (TableNotFoundException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  static boolean setAddress(Connector conn, String id, String expectedAddr, String newAddr) {
+    try (BatchWriter writer = conn.createBatchWriter("GothamPD", new BatchWriterConfig())) {
+      Mutation mutation = new Mutation(id);
+      mutation.put("location", "home", newAddr);
+      writer.addMutation(mutation);
+      return true;
+    } catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  public static Future<Void> modifyAddress(Connector conn, String id, Function<Stringreturn CompletableFuture.runAsync(() -> {
+      String currAddr, newAddr;
+      do {
+        currAddr = getAddress(conn, id);
+        newAddr = modifier.apply(currAddr);
+        System.out.printf("Thread %3d attempting change %20s -> %-20s\n",
+            Thread.currentThread().getId(), "'"+currAddr+"'", "'"+newAddr+"'");
+      } while (!setAddress(conn, id, currAddr, newAddr));
+    });
+  }
+
+  static void exercise(MiniAccumuloCluster mac) throws Exception {
+    Connector conn = mac.getConnector("root", "tourguide");
+    conn.tableOperations().create("GothamPD");
+
+    String id = "id0001";
+
+    setAddress(conn, id, null, "  1007 Mountain Drive, Gotham, New York  ");
+
+    // create async operation to trim whitespace
+    Future<Void> future1 = modifyAddress(conn, id, String::trim);
+
+    // create async operation to replace Dr with Drive
+    Future<Void> future2 = modifyAddress(conn, id, addr -> addr.replace(// create async operation to replace New York with NY
+    Future<Void> future3 = modifyAddress(conn, id, addr -> addr.replace(// wait for async operations to complete
+    future1.get();
+    future2.get();
+    future3.get();
+
+    // print the address stored in Accumulo
+    System.out.println("Final address : '"+getAddress(conn, id)+"'");
+  }
+
+
+ +

The following is one of a few possible outputs. Notice that only the +modification of Drive to Dr shows up in the final output. The other +modifications were lost.

+ +
Thread  36 attempting change '  1007 Mountain Drive, Gotham, New York  ' -> '1007 Mountain Drive, Gotham, New York'
+Thread  38 attempting change '  1007 Mountain Drive, Gotham, New York  ' -> '  1007 Mountain Drive, Gotham, NY  '
+Thread  37 attempting change '  1007 Mountain Drive, Gotham, New York  ' -> '  1007 Mountain Dr, Gotham, New York  '
+Final address : '  1007 Mountain Dr, Gotham, New York  '
+
+
+ +

To fix this example, make the following changes in setAddress() to use a +ConditionalWriter.

+ +
    +
  • Call createConditionalWriter instead of creating a batch writer
  • +
  • Create a Condition for the column ‘location:home’. If expectedAddr is not null, then call setValue passing expectedAddr. If [...] +
  • Replace Mutation with a ConditionalMutation and pass the condition to its constructor.
  • +
  • Call write passing it the conditional mutation.
  • +
  • Return true if getStatus from the Result returned by

    + < + - / 9 + 10 / 11 + + >

diff --git a/tour/data-model-code/index.html b/tour/data-model-code/index.html index 1fd5f6b..1527c6d 100644 --- a/tour/data-model-code/index.html +++ b/tour/data-model-code/index.html @@ -153,7 +153,7 @@

Accumulo Tour: Data Model Code

-

Tour page 4 of 9

+

Tour page 4 of 11

Below is the solution for the exercise.

@@ -234,7 +234,7 @@ if (e.keyCode == '39') { window.location = '/tour/authorizations/'; } < - 4 / 9 + 4 / 11 > diff --git a/tour/data-model/index.html b/tour/data-model/index.html index 9c27982..4bcdf41 100644 --- a/tour/data-model/index.html +++ b/tour/data-model/index.html @@ -153,7 +153,7 @@

Accumulo Tour: Data Model

-

Tour page 3 of 9

+

Tour page 3 of 11

Data is stored in Accumulo in a distributed sorted map. The Keys of the map are broken up logically into a few different parts, @@ -202,7 +202,7 @@ if (e.keyCode == '39') { window.location = '/tour/data-model-code/'; } < - 3 / 9 + 3 / 11 > diff --git a/tour/getting-started/index.html b/tour/getting-started/index.html index d61c5bd..732d789 100644 --- a/tour/getting-started/index.html +++ b/tour/getting-started/index.html @@ -153,7 +153,7 @@

Accumulo Tour: Getting Started

-

Tour page 1 of 9

+

Tour page 1 of 11

First make sure you have Java, Maven and Git installed on your machine. Oh you are already rocking? OK let’s go!

@@ -202,7 +202,7 @@ if (e.keyCode == '39') { window.location = '/tour/basic-read-write/'; }

- 1 / 9 + 1 / 11 > diff --git a/tour/index.html b/tour/index.html index a1dda02..81c3190 100644 --- a/tour/index.html +++ b/tour/index.html @@ -191,6 +191,12 @@ or create an issue

Batch Scanner Code

+
  • +

    Conditional Writer

    +
  • +
  • +

    Conditional Writer Code

    +
  • diff --git a/tour/ranges-splits/index.html b/tour/ranges-splits/index.html index a0f74d4..e70b03c 100644 --- a/tour/ranges-splits/index.html +++ b/tour/ranges-splits/index.html @@ -153,7 +153,7 @@

    Accumulo Tour: Ranges and Splits

    -

    Tour page 7 of 9

    +

    Tour page 7 of 11

    A Range is a specified group of Keys. There are many different ways to create a Range. Here are a few examples:

    @@ -212,7 +212,7 @@ if (e.keyCode == '39') { window.location = '/tour/batch-scanner/'; } < - 7 / 9 + 7 / 11 > diff --git a/tour/using-iterators/index.html b/tour/using-iterators/index.html index f40cf94..4a970f1 100644 --- a/tour/using-iterators/index.html +++ b/tour/using-iterators/index.html @@ -153,7 +153,7 @@

    Accumulo Tour: Using Iterators

    -

    Tour page of 9

    +

    Tour page of 11

    @@ -172,7 +172,7 @@ document.body.onkeyup = function(e){

    - / 9 + / 11

    -- To stop receiving notification emails like this one, please contact ['"commits@accumulo.apache.org" '].