Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id EC28E200C3D for ; Mon, 27 Feb 2017 17:24:38 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id EAC47160B56; Mon, 27 Feb 2017 16:24:38 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 65942160B7A for ; Mon, 27 Feb 2017 17:24:36 +0100 (CET) Received: (qmail 91243 invoked by uid 500); 27 Feb 2017 16:24:35 -0000 Mailing-List: contact commits-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list commits@accumulo.apache.org Received: (qmail 91204 invoked by uid 99); 27 Feb 2017 16:24:35 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Feb 2017 16:24:35 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 337A9DFE20; Mon, 27 Feb 2017 16:24:35 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: mjwall@apache.org To: commits@accumulo.apache.org Date: Mon, 27 Feb 2017 16:24:36 -0000 Message-Id: In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [02/16] accumulo-website git commit: Jekyll build from master:90a96b6 archived-at: Mon, 27 Feb 2017 16:24:39 -0000 http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/mapred.md ---------------------------------------------------------------------- diff --git a/1.8/examples/mapred.md b/1.8/examples/mapred.md new file mode 100644 index 0000000..9e9b17f --- /dev/null +++ b/1.8/examples/mapred.md @@ -0,0 +1,154 @@ +Title: Apache Accumulo MapReduce Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example uses mapreduce and accumulo to compute word counts for a set of +documents. This is accomplished using a map-only mapreduce job and a +accumulo table with combiners. + +To run this example you will need a directory in HDFS containing text files. +The accumulo readme will be used to show how to run this example. + + $ hadoop fs -copyFromLocal $ACCUMULO_HOME/README /user/username/wc/Accumulo.README + $ hadoop fs -ls /user/username/wc + Found 1 items + -rw-r--r-- 2 username supergroup 9359 2009-07-15 17:54 /user/username/wc/Accumulo.README + +The first part of running this example is to create a table with a combiner +for the column family count. + + $ ./bin/accumulo shell -u username -p password + Shell - Apache Accumulo Interactive Shell + - version: 1.5.0 + - instance name: instance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + username@instance> createtable wordCount + username@instance wordCount> setiter -class org.apache.accumulo.core.iterators.user.SummingCombiner -p 10 -t wordCount -majc -minc -scan + SummingCombiner interprets Values as Longs and adds them together. A variety of encodings (variable length, fixed length, or string) are available + ----------> set SummingCombiner parameter all, set to true to apply Combiner to every column, otherwise leave blank. if true, columns option will be ignored.: false + ----------> set SummingCombiner parameter columns, [:]{,[:]} escape non-alphanum chars using %.: count + ----------> set SummingCombiner parameter lossy, if true, failed decodes are ignored. Otherwise combiner will error on failed decodes (default false): : false + ----------> set SummingCombiner parameter type, : STRING + username@instance wordCount> quit + +After creating the table, run the word count map reduce job. + + $ bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.WordCount -i instance -z zookeepers --input /user/username/wc -t wordCount -u username -p password + + 11/02/07 18:20:11 INFO input.FileInputFormat: Total input paths to process : 1 + 11/02/07 18:20:12 INFO mapred.JobClient: Running job: job_201102071740_0003 + 11/02/07 18:20:13 INFO mapred.JobClient: map 0% reduce 0% + 11/02/07 18:20:20 INFO mapred.JobClient: map 100% reduce 0% + 11/02/07 18:20:22 INFO mapred.JobClient: Job complete: job_201102071740_0003 + 11/02/07 18:20:22 INFO mapred.JobClient: Counters: 6 + 11/02/07 18:20:22 INFO mapred.JobClient: Job Counters + 11/02/07 18:20:22 INFO mapred.JobClient: Launched map tasks=1 + 11/02/07 18:20:22 INFO mapred.JobClient: Data-local map tasks=1 + 11/02/07 18:20:22 INFO mapred.JobClient: FileSystemCounters + 11/02/07 18:20:22 INFO mapred.JobClient: HDFS_BYTES_READ=10487 + 11/02/07 18:20:22 INFO mapred.JobClient: Map-Reduce Framework + 11/02/07 18:20:22 INFO mapred.JobClient: Map input records=255 + 11/02/07 18:20:22 INFO mapred.JobClient: Spilled Records=0 + 11/02/07 18:20:22 INFO mapred.JobClient: Map output records=1452 + +After the map reduce job completes, query the accumulo table to see word +counts. + + $ ./bin/accumulo shell -u username -p password + username@instance> table wordCount + username@instance wordCount> scan -b the + the count:20080906 [] 75 + their count:20080906 [] 2 + them count:20080906 [] 1 + then count:20080906 [] 1 + there count:20080906 [] 1 + these count:20080906 [] 3 + this count:20080906 [] 6 + through count:20080906 [] 1 + time count:20080906 [] 3 + time. count:20080906 [] 1 + to count:20080906 [] 27 + total count:20080906 [] 1 + tserver, count:20080906 [] 1 + tserver.compaction.major.concurrent.max count:20080906 [] 1 + ... + +Another example to look at is +org.apache.accumulo.examples.simple.mapreduce.UniqueColumns. This example +computes the unique set of columns in a table and shows how a map reduce job +can directly read a tables files from HDFS. + +One more example available is +org.apache.accumulo.examples.simple.mapreduce.TokenFileWordCount. +The TokenFileWordCount example works exactly the same as the WordCount example +explained above except that it uses a token file rather than giving the +password directly to the map-reduce job (this avoids having the password +displayed in the job's configuration which is world-readable). + +To create a token file, use the create-token utility + + $ ./bin/accumulo create-token + +It defaults to creating a PasswordToken, but you can specify the token class +with -tc (requires the fully qualified class name). Based on the token class, +it will prompt you for each property required to create the token. + +The last value it prompts for is a local filename to save to. If this file +exists, it will append the new token to the end. Multiple tokens can exist in +a file, but only the first one for each user will be recognized. + +Rather than waiting for the prompts, you can specify some options when calling +create-token, for example + + $ ./bin/accumulo create-token -u root -p secret -f root.pw + +would create a token file containing a PasswordToken for +user 'root' with password 'secret' and saved to 'root.pw' + +This local file needs to be uploaded to hdfs to be used with the +map-reduce job. For example, if the file were 'root.pw' in the local directory: + + $ hadoop fs -put root.pw root.pw + +This would put 'root.pw' in the user's home directory in hdfs. + +Because the basic WordCount example uses Opts to parse its arguments +(which extends ClientOnRequiredTable), you can use a token file with +the basic WordCount example by calling the same command as explained above +except replacing the password with the token file (rather than -p, use -tf). + + $ ./bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.WordCount -i instance -z zookeepers --input /user/username/wc -t wordCount -u username -tf tokenfile + +In the above examples, username was 'root' and tokenfile was 'root.pw' + +However, if you don't want to use the Opts class to parse arguments, +the TokenFileWordCount is an example of using the token file manually. + + $ bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.TokenFileWordCount instance zookeepers username tokenfile /user/username/wc wordCount + +The results should be the same as the WordCount example except that the +authentication token was not stored in the configuration. It was instead +stored in a file that the map-reduce job pulled into the distributed cache. +(If you ran either of these on the same table right after the +WordCount example, then the resulting counts should just double.) + + + + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/maxmutation.html ---------------------------------------------------------------------- diff --git a/1.8/examples/maxmutation.html b/1.8/examples/maxmutation.html index 5c491a1..a5dee62 100644 --- a/1.8/examples/maxmutation.html +++ b/1.8/examples/maxmutation.html @@ -25,7 +25,7 @@ -Apache Accumulo MaxMutation Constraints Example +Apache Accumulo™ @@ -84,7 +84,7 @@
  • Releases
      -
    • 1.8.0 (Latest)
      • +
    • 1.8.1 (Latest)
    • 1.7.2
    • 1.6.6
    • Archive
      • @@ -140,9 +140,27 @@
      -

      Apache Accumulo MaxMutation Constraints Example

      +

      Apache Accumulo™

      -

      This an example of how to limit the size of mutations that will be accepted into +

      Title: Apache Accumulo MaxMutation Constraints Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      This an example of how to limit the size of mutations that will be accepted into a table. Under the default configuration, accumulo does not provide a limitation on the size of mutations that can be ingested. Poorly behaved writers might inadvertently create mutations so large, that they cause the tablet servers to http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/maxmutation.md ---------------------------------------------------------------------- diff --git a/1.8/examples/maxmutation.md b/1.8/examples/maxmutation.md new file mode 100644 index 0000000..45b80d4 --- /dev/null +++ b/1.8/examples/maxmutation.md @@ -0,0 +1,49 @@ +Title: Apache Accumulo MaxMutation Constraints Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This an example of how to limit the size of mutations that will be accepted into +a table. Under the default configuration, accumulo does not provide a limitation +on the size of mutations that can be ingested. Poorly behaved writers might +inadvertently create mutations so large, that they cause the tablet servers to +run out of memory. A simple contraint can be added to a table to reject very +large mutations. + + $ ./bin/accumulo shell -u username -p password + + Shell - Apache Accumulo Interactive Shell + - + - version: 1.5.0 + - instance name: instance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + username@instance> createtable test_ingest + username@instance test_ingest> config -t test_ingest -s table.constraint.1=org.apache.accumulo.examples.simple.constraints.MaxMutationSize + username@instance test_ingest> + + +Now the table will reject any mutation that is larger than 1/256th of the +working memory of the tablet server. The following command attempts to ingest +a single row with 10000 columns, which exceeds the memory limit. Depending on the +amount of Java heap your tserver(s) are given, you may have to increase the number +of columns provided to see the failure. + + $ ./bin/accumulo org.apache.accumulo.test.TestIngest -i instance -z zookeepers -u username -p password --rows 1 --cols 10000 + ERROR : Constraint violates : ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.simple.constraints.MaxMutationSize, violationCode:0, violationDescription:mutation exceeded maximum size of 188160, numberOfViolatingMutations:1) + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/regex.html ---------------------------------------------------------------------- diff --git a/1.8/examples/regex.html b/1.8/examples/regex.html index 143b636..309d9d7 100644 --- a/1.8/examples/regex.html +++ b/1.8/examples/regex.html @@ -25,7 +25,7 @@ -Apache Accumulo Regex Example +Apache Accumulo™ @@ -84,7 +84,7 @@

    • Releases
      • -

      Apache Accumulo Regex Example

      +

      Apache Accumulo™

      -

      This example uses mapreduce and accumulo to find items using regular expressions. +

      Title: Apache Accumulo Regex Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      This example uses mapreduce and accumulo to find items using regular expressions. This is accomplished using a map-only mapreduce job and a scan-time iterator.

      To run this example you will need some data in a table. The following will http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/regex.md ---------------------------------------------------------------------- diff --git a/1.8/examples/regex.md b/1.8/examples/regex.md new file mode 100644 index 0000000..ea9f208 --- /dev/null +++ b/1.8/examples/regex.md @@ -0,0 +1,57 @@ +Title: Apache Accumulo Regex Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example uses mapreduce and accumulo to find items using regular expressions. +This is accomplished using a map-only mapreduce job and a scan-time iterator. + +To run this example you will need some data in a table. The following will +put a trivial amount of data into accumulo using the accumulo shell: + + $ ./bin/accumulo shell -u username -p password + Shell - Apache Accumulo Interactive Shell + - version: 1.5.0 + - instance name: instance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + username@instance> createtable input + username@instance> insert dogrow dogcf dogcq dogvalue + username@instance> insert catrow catcf catcq catvalue + username@instance> quit + +The RegexExample class sets an iterator on the scanner. This does pattern matching +against each key/value in accumulo, and only returns matching items. It will do this +in parallel and will store the results in files in hdfs. + +The following will search for any rows in the input table that starts with "dog": + + $ bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.RegexExample -u user -p passwd -i instance -t input --rowRegex 'dog.*' --output /tmp/output + + $ hadoop fs -ls /tmp/output + Found 3 items + -rw-r--r-- 1 username supergroup 0 2013-01-10 14:11 /tmp/output/_SUCCESS + drwxr-xr-x - username supergroup 0 2013-01-10 14:10 /tmp/output/_logs + -rw-r--r-- 1 username supergroup 51 2013-01-10 14:10 /tmp/output/part-m-00000 + +We can see the output of our little map-reduce job: + + $ hadoop fs -text /tmp/output/part-m-00000 + dogrow dogcf:dogcq [] 1357844987994 false dogvalue + + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/reservations.html ---------------------------------------------------------------------- diff --git a/1.8/examples/reservations.html b/1.8/examples/reservations.html index 4bfdead..ecf53db 100644 --- a/1.8/examples/reservations.html +++ b/1.8/examples/reservations.html @@ -25,7 +25,7 @@ -Apache Accumulo Isolation Example +Apache Accumulo™ @@ -84,7 +84,7 @@

    • Releases
      • -

      Apache Accumulo Isolation Example

      +

      Apache Accumulo™

      -

      This example shows running a simple reservation system implemented using +

      Title: Apache Accumulo Isolation Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      This example shows running a simple reservation system implemented using conditional mutations. This system guarantees that only one concurrent user can reserve a resource. The example’s reserve command allows multiple users to be specified. When this is done, it creates a separate reservation thread for each http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/reservations.md ---------------------------------------------------------------------- diff --git a/1.8/examples/reservations.md b/1.8/examples/reservations.md new file mode 100644 index 0000000..ff111b4 --- /dev/null +++ b/1.8/examples/reservations.md @@ -0,0 +1,66 @@ +Title: Apache Accumulo Isolation Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example shows running a simple reservation system implemented using +conditional mutations. This system guarantees that only one concurrent user can +reserve a resource. The example's reserve command allows multiple users to be +specified. When this is done, it creates a separate reservation thread for each +user. In the example below threads are spun up for alice, bob, eve, mallory, +and trent to reserve room06 on 20140101. Bob ends up getting the reservation +and everyone else is put on a wait list. The example code will take any string +for what, when and who. + + $ ./bin/accumulo org.apache.accumulo.examples.simple.reservations.ARS + >connect test16 localhost root secret ars + connected + > + Commands : + reserve {who} + cancel + list + >reserve room06 20140101 alice bob eve mallory trent + bob : RESERVED + mallory : WAIT_LISTED + alice : WAIT_LISTED + trent : WAIT_LISTED + eve : WAIT_LISTED + >list room06 20140101 + Reservation holder : bob + Wait list : [mallory, alice, trent, eve] + >cancel room06 20140101 alice + >cancel room06 20140101 bob + >list room06 20140101 + Reservation holder : mallory + Wait list : [trent, eve] + >quit + +Scanning the table in the Accumulo shell after running the example shows the +following: + + root@test16> table ars + root@test16 ars> scan + room06:20140101 res:0001 [] mallory + room06:20140101 res:0003 [] trent + room06:20140101 res:0004 [] eve + room06:20140101 tx:seq [] 6 + +The tx:seq column is incremented for each update to the row allowing for +detection of concurrent changes. For an update to go through, the sequence +number must not have changed since the data was read. If it does change, +the conditional mutation will fail and the example code will retry. + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/rgbalancer.html ---------------------------------------------------------------------- diff --git a/1.8/examples/rgbalancer.html b/1.8/examples/rgbalancer.html index f4f65d5..80b11f1 100644 --- a/1.8/examples/rgbalancer.html +++ b/1.8/examples/rgbalancer.html @@ -25,7 +25,7 @@ -Apache Accumulo Hello World Example +Apache Accumulo™ @@ -84,7 +84,7 @@

    • Releases
      • -

      Apache Accumulo Hello World Example

      +

      Apache Accumulo™

      -

      For some data access patterns, its important to spread groups of tablets within +

      Title: Apache Accumulo Hello World Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      For some data access patterns, its important to spread groups of tablets within a table out evenly. Accumulo has a balancer that can do this using a regular expression to group tablets. This example shows how this balancer spreads 4 groups of tablets within a table evenly across 17 tablet servers.

      http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/rgbalancer.md ---------------------------------------------------------------------- diff --git a/1.8/examples/rgbalancer.md b/1.8/examples/rgbalancer.md new file mode 100644 index 0000000..f192a93 --- /dev/null +++ b/1.8/examples/rgbalancer.md @@ -0,0 +1,159 @@ +Title: Apache Accumulo Hello World Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +For some data access patterns, its important to spread groups of tablets within +a table out evenly. Accumulo has a balancer that can do this using a regular +expression to group tablets. This example shows how this balancer spreads 4 +groups of tablets within a table evenly across 17 tablet servers. + +Below shows creating a table and adding splits. For this example we would like +all of the tablets where the split point has the same two digits to be on +different tservers. This gives us four groups of tablets: 01, 02, 03, and 04. + + root@accumulo> createtable testRGB + root@accumulo testRGB> addsplits -t testRGB 01b 01m 01r 01z 02b 02m 02r 02z 03b 03m 03r 03z 04a 04b 04c 04d 04e 04f 04g 04h 04i 04j 04k 04l 04m 04n 04o 04p + root@accumulo testRGB> tables -l + accumulo.metadata => !0 + accumulo.replication => +rep + accumulo.root => +r + testRGB => 2 + trace => 1 + +After adding the splits we look at the locations in the metadata table. + + root@accumulo testRGB> scan -t accumulo.metadata -b 2; -e 2< -c loc + 2;01b loc:34a5f6e086b000c [] ip-10-1-2-25:9997 + 2;01m loc:34a5f6e086b000c [] ip-10-1-2-25:9997 + 2;01r loc:14a5f6e079d0011 [] ip-10-1-2-15:9997 + 2;01z loc:14a5f6e079d000f [] ip-10-1-2-13:9997 + 2;02b loc:34a5f6e086b000b [] ip-10-1-2-26:9997 + 2;02m loc:14a5f6e079d000c [] ip-10-1-2-28:9997 + 2;02r loc:14a5f6e079d0012 [] ip-10-1-2-27:9997 + 2;02z loc:14a5f6e079d0012 [] ip-10-1-2-27:9997 + 2;03b loc:14a5f6e079d000d [] ip-10-1-2-21:9997 + 2;03m loc:14a5f6e079d000e [] ip-10-1-2-20:9997 + 2;03r loc:14a5f6e079d000d [] ip-10-1-2-21:9997 + 2;03z loc:14a5f6e079d000e [] ip-10-1-2-20:9997 + 2;04a loc:34a5f6e086b000b [] ip-10-1-2-26:9997 + 2;04b loc:14a5f6e079d0010 [] ip-10-1-2-17:9997 + 2;04c loc:14a5f6e079d0010 [] ip-10-1-2-17:9997 + 2;04d loc:24a5f6e07d3000c [] ip-10-1-2-16:9997 + 2;04e loc:24a5f6e07d3000d [] ip-10-1-2-29:9997 + 2;04f loc:24a5f6e07d3000c [] ip-10-1-2-16:9997 + 2;04g loc:24a5f6e07d3000a [] ip-10-1-2-14:9997 + 2;04h loc:14a5f6e079d000c [] ip-10-1-2-28:9997 + 2;04i loc:34a5f6e086b000d [] ip-10-1-2-19:9997 + 2;04j loc:34a5f6e086b000d [] ip-10-1-2-19:9997 + 2;04k loc:24a5f6e07d30009 [] ip-10-1-2-23:9997 + 2;04l loc:24a5f6e07d3000b [] ip-10-1-2-22:9997 + 2;04m loc:24a5f6e07d30009 [] ip-10-1-2-23:9997 + 2;04n loc:24a5f6e07d3000b [] ip-10-1-2-22:9997 + 2;04o loc:34a5f6e086b000a [] ip-10-1-2-18:9997 + 2;04p loc:24a5f6e07d30008 [] ip-10-1-2-24:9997 + 2< loc:24a5f6e07d30008 [] ip-10-1-2-24:9997 + +Below the information above was massaged to show which tablet groups are on +each tserver. The four tablets in group 03 are on two tservers, ideally those +tablets would be spread across 4 tservers. Note the default tablet (2<) was +categorized as group 04 below. + + ip-10-1-2-13:9997 01 + ip-10-1-2-14:9997 04 + ip-10-1-2-15:9997 01 + ip-10-1-2-16:9997 04 04 + ip-10-1-2-17:9997 04 04 + ip-10-1-2-18:9997 04 + ip-10-1-2-19:9997 04 04 + ip-10-1-2-20:9997 03 03 + ip-10-1-2-21:9997 03 03 + ip-10-1-2-22:9997 04 04 + ip-10-1-2-23:9997 04 04 + ip-10-1-2-24:9997 04 04 + ip-10-1-2-25:9997 01 01 + ip-10-1-2-26:9997 02 04 + ip-10-1-2-27:9997 02 02 + ip-10-1-2-28:9997 02 04 + ip-10-1-2-29:9997 04 + +To remedy this situation, the RegexGroupBalancer is configured with the +commands below. The configured regular expression selects the first two digits +from a tablets end row as the group id. Tablets that don't match and the +default tablet are configured to be in group 04. + + root@accumulo testRGB> config -t testRGB -s table.custom.balancer.group.regex.pattern=(\\d\\d).* + root@accumulo testRGB> config -t testRGB -s table.custom.balancer.group.regex.default=04 + root@accumulo testRGB> config -t testRGB -s table.balancer=org.apache.accumulo.server.master.balancer.RegexGroupBalancer + +After waiting a little bit, look at the tablet locations again and all is good. + + root@accumulo testRGB> scan -t accumulo.metadata -b 2; -e 2< -c loc + 2;01b loc:34a5f6e086b000a [] ip-10-1-2-18:9997 + 2;01m loc:34a5f6e086b000c [] ip-10-1-2-25:9997 + 2;01r loc:14a5f6e079d0011 [] ip-10-1-2-15:9997 + 2;01z loc:14a5f6e079d000f [] ip-10-1-2-13:9997 + 2;02b loc:34a5f6e086b000b [] ip-10-1-2-26:9997 + 2;02m loc:14a5f6e079d000c [] ip-10-1-2-28:9997 + 2;02r loc:34a5f6e086b000d [] ip-10-1-2-19:9997 + 2;02z loc:14a5f6e079d0012 [] ip-10-1-2-27:9997 + 2;03b loc:24a5f6e07d3000d [] ip-10-1-2-29:9997 + 2;03m loc:24a5f6e07d30009 [] ip-10-1-2-23:9997 + 2;03r loc:14a5f6e079d000d [] ip-10-1-2-21:9997 + 2;03z loc:14a5f6e079d000e [] ip-10-1-2-20:9997 + 2;04a loc:34a5f6e086b000b [] ip-10-1-2-26:9997 + 2;04b loc:34a5f6e086b000c [] ip-10-1-2-25:9997 + 2;04c loc:14a5f6e079d0010 [] ip-10-1-2-17:9997 + 2;04d loc:14a5f6e079d000e [] ip-10-1-2-20:9997 + 2;04e loc:24a5f6e07d3000d [] ip-10-1-2-29:9997 + 2;04f loc:24a5f6e07d3000c [] ip-10-1-2-16:9997 + 2;04g loc:24a5f6e07d3000a [] ip-10-1-2-14:9997 + 2;04h loc:14a5f6e079d000c [] ip-10-1-2-28:9997 + 2;04i loc:14a5f6e079d0011 [] ip-10-1-2-15:9997 + 2;04j loc:34a5f6e086b000d [] ip-10-1-2-19:9997 + 2;04k loc:14a5f6e079d0012 [] ip-10-1-2-27:9997 + 2;04l loc:14a5f6e079d000f [] ip-10-1-2-13:9997 + 2;04m loc:24a5f6e07d30009 [] ip-10-1-2-23:9997 + 2;04n loc:24a5f6e07d3000b [] ip-10-1-2-22:9997 + 2;04o loc:34a5f6e086b000a [] ip-10-1-2-18:9997 + 2;04p loc:14a5f6e079d000d [] ip-10-1-2-21:9997 + 2< loc:24a5f6e07d30008 [] ip-10-1-2-24:9997 + +Once again, the data above is transformed to make it easier to see which groups +are on tservers. The transformed data below shows that all groups are now +evenly spread. + + ip-10-1-2-13:9997 01 04 + ip-10-1-2-14:9997 04 + ip-10-1-2-15:9997 01 04 + ip-10-1-2-16:9997 04 + ip-10-1-2-17:9997 04 + ip-10-1-2-18:9997 01 04 + ip-10-1-2-19:9997 02 04 + ip-10-1-2-20:9997 03 04 + ip-10-1-2-21:9997 03 04 + ip-10-1-2-22:9997 04 + ip-10-1-2-23:9997 03 04 + ip-10-1-2-24:9997 04 + ip-10-1-2-25:9997 01 04 + ip-10-1-2-26:9997 02 04 + ip-10-1-2-27:9997 02 04 + ip-10-1-2-28:9997 02 04 + ip-10-1-2-29:9997 03 04 + +If you need this functionality, but a regular expression does not meet your +needs then extend GroupBalancer. This allows you to specify a partitioning +function in Java. Use the RegexGroupBalancer source as an example. http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/rowhash.html ---------------------------------------------------------------------- diff --git a/1.8/examples/rowhash.html b/1.8/examples/rowhash.html index f6d0d85..7fed3c0 100644 --- a/1.8/examples/rowhash.html +++ b/1.8/examples/rowhash.html @@ -25,7 +25,7 @@ -Apache Accumulo RowHash Example +Apache Accumulo™ @@ -84,7 +84,7 @@
    • Releases
      • -

      Apache Accumulo RowHash Example

      +

      Apache Accumulo™

      -

      This example shows a simple map/reduce job that reads from an accumulo table and +

      Title: Apache Accumulo RowHash Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      This example shows a simple map/reduce job that reads from an accumulo table and writes back into that table.

      To run this example you will need some data in a table. The following will http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/rowhash.md ---------------------------------------------------------------------- diff --git a/1.8/examples/rowhash.md b/1.8/examples/rowhash.md new file mode 100644 index 0000000..43782c9 --- /dev/null +++ b/1.8/examples/rowhash.md @@ -0,0 +1,59 @@ +Title: Apache Accumulo RowHash Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example shows a simple map/reduce job that reads from an accumulo table and +writes back into that table. + +To run this example you will need some data in a table. The following will +put a trivial amount of data into accumulo using the accumulo shell: + + $ ./bin/accumulo shell -u username -p password + Shell - Apache Accumulo Interactive Shell + - version: 1.5.0 + - instance name: instance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + username@instance> createtable input + username@instance> insert a-row cf cq value + username@instance> insert b-row cf cq value + username@instance> quit + +The RowHash class will insert a hash for each row in the database if it contains a +specified colum. Here's how you run the map/reduce job + + $ bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.RowHash -u user -p passwd -i instance -t input --column cf:cq + +Now we can scan the table and see the hashes: + + $ ./bin/accumulo shell -u username -p password + Shell - Apache Accumulo Interactive Shell + - version: 1.5.0 + - instance name: instance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + username@instance> scan -t input + a-row cf:cq [] value + a-row cf-HASHTYPE:cq-MD5BASE64 [] IGPBYI1uC6+AJJxC4r5YBA== + b-row cf:cq [] value + b-row cf-HASHTYPE:cq-MD5BASE64 [] IGPBYI1uC6+AJJxC4r5YBA== + username@instance> + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/sample.html ---------------------------------------------------------------------- diff --git a/1.8/examples/sample.html b/1.8/examples/sample.html index 71bb74e..3273ba0 100644 --- a/1.8/examples/sample.html +++ b/1.8/examples/sample.html @@ -25,7 +25,7 @@ -Apache Accumulo Batch Writing and Scanning Example +Apache Accumulo™ @@ -84,7 +84,7 @@

    • Releases
      • -

      Apache Accumulo Batch Writing and Scanning Example

      +

      Apache Accumulo™

      -

      Basic Sampling Example

      +

      Title: Apache Accumulo Batch Writing and Scanning Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      Basic Sampling Example

      Accumulo supports building a set of sample data that can be efficiently accessed by scanners. What data is included in the sample set is configurable. http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/sample.md ---------------------------------------------------------------------- diff --git a/1.8/examples/sample.md b/1.8/examples/sample.md new file mode 100644 index 0000000..3642cc6 --- /dev/null +++ b/1.8/examples/sample.md @@ -0,0 +1,192 @@ +Title: Apache Accumulo Batch Writing and Scanning Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Basic Sampling Example +---------------------- + +Accumulo supports building a set of sample data that can be efficiently +accessed by scanners. What data is included in the sample set is configurable. +Below, some data representing documents are inserted. + + root@instance sampex> createtable sampex + root@instance sampex> insert 9255 doc content 'abcde' + root@instance sampex> insert 9255 doc url file://foo.txt + root@instance sampex> insert 8934 doc content 'accumulo scales' + root@instance sampex> insert 8934 doc url file://accumulo_notes.txt + root@instance sampex> insert 2317 doc content 'milk, eggs, bread, parmigiano-reggiano' + root@instance sampex> insert 2317 doc url file://groceries/9.txt + root@instance sampex> insert 3900 doc content 'EC2 ate my homework' + root@instance sampex> insert 3900 doc uril file://final_project.txt + +Below the table sampex is configured to build a sample set. The configuration +causes Accumulo to include any row where `murmur3_32(row) % 3 ==0` in the +tables sample data. + + root@instance sampex> config -t sampex -s table.sampler.opt.hasher=murmur3_32 + root@instance sampex> config -t sampex -s table.sampler.opt.modulus=3 + root@instance sampex> config -t sampex -s table.sampler=org.apache.accumulo.core.client.sample.RowSampler + +Below, attempting to scan the sample returns an error. This is because data +was inserted before the sample set was configured. + + root@instance sampex> scan --sample + 2015-09-09 12:21:50,643 [shell.Shell] ERROR: org.apache.accumulo.core.client.SampleNotPresentException: Table sampex(ID:2) does not have sampling configured or built + +To remedy this problem, the following command will flush in memory data and +compact any files that do not contain the correct sample data. + + root@instance sampex> compact -t sampex --sf-no-sample + +After the compaction, the sample scan works. + + root@instance sampex> scan --sample + 2317 doc:content [] milk, eggs, bread, parmigiano-reggiano + 2317 doc:url [] file://groceries/9.txt + +The commands below show that updates to data in the sample are seen when +scanning the sample. + + root@instance sampex> insert 2317 doc content 'milk, eggs, bread, parmigiano-reggiano, butter' + root@instance sampex> scan --sample + 2317 doc:content [] milk, eggs, bread, parmigiano-reggiano, butter + 2317 doc:url [] file://groceries/9.txt + +Inorder to make scanning the sample fast, sample data is partitioned as data is +written to Accumulo. This means if the sample configuration is changed, that +data written previously is partitioned using a different criteria. Accumulo +will detect this situation and fail sample scans. The commands below show this +failure and fixiing the problem with a compaction. + + root@instance sampex> config -t sampex -s table.sampler.opt.modulus=2 + root@instance sampex> scan --sample + 2015-09-09 12:22:51,058 [shell.Shell] ERROR: org.apache.accumulo.core.client.SampleNotPresentException: Table sampex(ID:2) does not have sampling configured or built + root@instance sampex> compact -t sampex --sf-no-sample + 2015-09-09 12:23:07,242 [shell.Shell] INFO : Compaction of table sampex started for given range + root@instance sampex> scan --sample + 2317 doc:content [] milk, eggs, bread, parmigiano-reggiano + 2317 doc:url [] file://groceries/9.txt + 3900 doc:content [] EC2 ate my homework + 3900 doc:uril [] file://final_project.txt + 9255 doc:content [] abcde + 9255 doc:url [] file://foo.txt + +The example above is replicated in a java program using the Accumulo API. +Below is the program name and the command to run it. + + ./bin/accumulo org.apache.accumulo.examples.simple.sample.SampleExample -i instance -z localhost -u root -p secret + +The commands below look under the hood to give some insight into how this +feature works. The commands determine what files the sampex table is using. + + root@instance sampex> tables -l + accumulo.metadata => !0 + accumulo.replication => +rep + accumulo.root => +r + sampex => 2 + trace => 1 + root@instance sampex> scan -t accumulo.metadata -c file -b 2 -e 2< + 2< file:hdfs://localhost:10000/accumulo/tables/2/default_tablet/A000000s.rf [] 702,8 + +Below shows running `accumulo rfile-info` on the file above. This shows the +rfile has a normal default locality group and a sample default locality group. +The output also shows the configuration used to create the sample locality +group. The sample configuration within a rfile must match the tables sample +configuration for sample scan to work. + + $ ./bin/accumulo rfile-info hdfs://localhost:10000/accumulo/tables/2/default_tablet/A000000s.rf + Reading file: hdfs://localhost:10000/accumulo/tables/2/default_tablet/A000000s.rf + RFile Version : 8 + + Locality group : + Start block : 0 + Num blocks : 1 + Index level 0 : 35 bytes 1 blocks + First key : 2317 doc:content [] 1437672014986 false + Last key : 9255 doc:url [] 1437672014875 false + Num entries : 8 + Column families : [doc] + + Sample Configuration : + Sampler class : org.apache.accumulo.core.client.sample.RowSampler + Sampler options : {hasher=murmur3_32, modulus=2} + + Sample Locality group : + Start block : 0 + Num blocks : 1 + Index level 0 : 36 bytes 1 blocks + First key : 2317 doc:content [] 1437672014986 false + Last key : 9255 doc:url [] 1437672014875 false + Num entries : 6 + Column families : [doc] + + Meta block : BCFile.index + Raw size : 4 bytes + Compressed size : 12 bytes + Compression type : gz + + Meta block : RFile.index + Raw size : 309 bytes + Compressed size : 176 bytes + Compression type : gz + + +Shard Sampling Example +------------------------- + +`README.shard` shows how to index and search files using Accumulo. That +example indexes documents into a table named `shard`. The indexing scheme used +in that example places the document name in the column qualifier. A useful +sample of this indexing scheme should contain all data for any document in the +sample. To accomplish this, the following commands build a sample for the +shard table based on the column qualifier. + + root@instance shard> config -t shard -s table.sampler.opt.hasher=murmur3_32 + root@instance shard> config -t shard -s table.sampler.opt.modulus=101 + root@instance shard> config -t shard -s table.sampler.opt.qualifier=true + root@instance shard> config -t shard -s table.sampler=org.apache.accumulo.core.client.sample.RowColumnSampler + root@instance shard> compact -t shard --sf-no-sample -w + 2015-07-23 15:00:09,280 [shell.Shell] INFO : Compacting table ... + 2015-07-23 15:00:10,134 [shell.Shell] INFO : Compaction of table shard completed for given range + +After enabling sampling, the command below counts the number of documents in +the sample containing the words `import` and `int`. + + $ ./bin/accumulo org.apache.accumulo.examples.simple.shard.Query --sample -i instance16 -z localhost -t shard -u root -p secret import int | fgrep '.java' | wc + 11 11 1246 + +The command below counts the total number of documents containing the words +`import` and `int`. + + $ ./bin/accumulo org.apache.accumulo.examples.simple.shard.Query -i instance16 -z localhost -t shard -u root -p secret import int | fgrep '.java' | wc + 1085 1085 118175 + +The counts 11 out of 1085 total are around what would be expected for a modulus +of 101. Querying the sample first provides a quick way to estimate how much data +the real query will bring back. + +Another way sample data could be used with the shard example is with a +specialized iterator. In the examples source code there is an iterator named +CutoffIntersectingIterator. This iterator first checks how many documents are +found in the sample data. If too many documents are found in the sample data, +then it returns nothing. Otherwise it proceeds to query the full data set. +To experiment with this iterator, use the following command. The +`--sampleCutoff` option below will cause the query to return nothing if based +on the sample it appears a query would return more than 1000 documents. + + $ ./bin/accumulo org.apache.accumulo.examples.simple.shard.Query --sampleCutoff 1000 -i instance16 -z localhost -t shard -u root -p secret import int | fgrep '.java' | wc http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/shard.html ---------------------------------------------------------------------- diff --git a/1.8/examples/shard.html b/1.8/examples/shard.html index 3508e01..40c7576 100644 --- a/1.8/examples/shard.html +++ b/1.8/examples/shard.html @@ -25,7 +25,7 @@ -Apache Accumulo Shard Example +Apache Accumulo™ @@ -84,7 +84,7 @@

    • Releases
      • -

      Apache Accumulo Shard Example

      +

      Apache Accumulo™

      -

      Accumulo has an iterator called the intersecting iterator which supports querying a term index that is partitioned by +

      Title: Apache Accumulo Shard Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      Accumulo has an iterator called the intersecting iterator which supports querying a term index that is partitioned by document, or “sharded”. This example shows how to use the intersecting iterator through these four programs:

        http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/shard.md ---------------------------------------------------------------------- diff --git a/1.8/examples/shard.md b/1.8/examples/shard.md new file mode 100644 index 0000000..d08658a --- /dev/null +++ b/1.8/examples/shard.md @@ -0,0 +1,67 @@ +Title: Apache Accumulo Shard Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Accumulo has an iterator called the intersecting iterator which supports querying a term index that is partitioned by +document, or "sharded". This example shows how to use the intersecting iterator through these four programs: + + * Index.java - Indexes a set of text files into an Accumulo table + * Query.java - Finds documents containing a given set of terms. + * Reverse.java - Reads the index table and writes a map of documents to terms into another table. + * ContinuousQuery.java Uses the table populated by Reverse.java to select N random terms per document. Then it continuously and randomly queries those terms. + +To run these example programs, create two tables like below. + + username@instance> createtable shard + username@instance shard> createtable doc2term + +After creating the tables, index some files. The following command indexes all of the java files in the Accumulo source code. + + $ cd /local/username/workspace/accumulo/ + $ find core/src server/src -name "*.java" | xargs ./bin/accumulo org.apache.accumulo.examples.simple.shard.Index -i instance -z zookeepers -t shard -u username -p password --partitions 30 + +The following command queries the index to find all files containing 'foo' and 'bar'. + + $ cd $ACCUMULO_HOME + $ ./bin/accumulo org.apache.accumulo.examples.simple.shard.Query -i instance -z zookeepers -t shard -u username -p password foo bar + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/security/ColumnVisibilityTest.java + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/client/mock/MockConnectorTest.java + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/security/VisibilityEvaluatorTest.java + /local/username/workspace/accumulo/src/server/src/main/java/accumulo/test/functional/RowDeleteTest.java + /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/logger/TestLogWriter.java + /local/username/workspace/accumulo/src/server/src/main/java/accumulo/test/functional/DeleteEverythingTest.java + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/data/KeyExtentTest.java + /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/constraints/MetadataConstraintsTest.java + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/iterators/WholeRowIteratorTest.java + /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/util/DefaultMapTest.java + /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/tabletserver/InMemoryMapTest.java + +In order to run ContinuousQuery, we need to run Reverse.java to populate doc2term. + + $ ./bin/accumulo org.apache.accumulo.examples.simple.shard.Reverse -i instance -z zookeepers --shardTable shard --doc2Term doc2term -u username -p password + +Below ContinuousQuery is run using 5 terms. So it selects 5 random terms from each document, then it continually +randomly selects one set of 5 terms and queries. It prints the number of matching documents and the time in seconds. + + $ ./bin/accumulo org.apache.accumulo.examples.simple.shard.ContinuousQuery -i instance -z zookeepers --shardTable shard --doc2Term doc2term -u username -p password --terms 5 + [public, core, class, binarycomparable, b] 2 0.081 + [wordtodelete, unindexdocument, doctablename, putdelete, insert] 1 0.041 + [import, columnvisibilityinterpreterfactory, illegalstateexception, cv, columnvisibility] 1 0.049 + [getpackage, testversion, util, version, 55] 1 0.048 + [for, static, println, public, the] 55 0.211 + [sleeptime, wrappingiterator, options, long, utilwaitthread] 1 0.057 + [string, public, long, 0, wait] 12 0.132 http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/tabletofile.html ---------------------------------------------------------------------- diff --git a/1.8/examples/tabletofile.html b/1.8/examples/tabletofile.html index b9c8b3c..b4bac0d 100644 --- a/1.8/examples/tabletofile.html +++ b/1.8/examples/tabletofile.html @@ -25,7 +25,7 @@ -Apache Accumulo Table-to-File Example +Apache Accumulo™ @@ -84,7 +84,7 @@
      • Releases
      -

      Apache Accumulo Table-to-File Example

      +

      Apache Accumulo™

      -

      This example uses mapreduce to extract specified columns from an existing table.

      +

      Title: Apache Accumulo Table-to-File Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      This example uses mapreduce to extract specified columns from an existing table.

      To run this example you will need some data in a table. The following will put a trivial amount of data into accumulo using the accumulo shell:

      http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/tabletofile.md ---------------------------------------------------------------------- diff --git a/1.8/examples/tabletofile.md b/1.8/examples/tabletofile.md new file mode 100644 index 0000000..08b7cc9 --- /dev/null +++ b/1.8/examples/tabletofile.md @@ -0,0 +1,59 @@ +Title: Apache Accumulo Table-to-File Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example uses mapreduce to extract specified columns from an existing table. + +To run this example you will need some data in a table. The following will +put a trivial amount of data into accumulo using the accumulo shell: + + $ ./bin/accumulo shell -u username -p password + Shell - Apache Accumulo Interactive Shell + - version: 1.5.0 + - instance name: instance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + username@instance> createtable input + username@instance> insert dog cf cq dogvalue + username@instance> insert cat cf cq catvalue + username@instance> insert junk family qualifier junkvalue + username@instance> quit + +The TableToFile class configures a map-only job to read the specified columns and +write the key/value pairs to a file in HDFS. + +The following will extract the rows containing the column "cf:cq": + + $ bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.TableToFile -u user -p passwd -i instance -t input --columns cf:cq --output /tmp/output + + $ hadoop fs -ls /tmp/output + -rw-r--r-- 1 username supergroup 0 2013-01-10 14:44 /tmp/output/_SUCCESS + drwxr-xr-x - username supergroup 0 2013-01-10 14:44 /tmp/output/_logs + drwxr-xr-x - username supergroup 0 2013-01-10 14:44 /tmp/output/_logs/history + -rw-r--r-- 1 username supergroup 9049 2013-01-10 14:44 /tmp/output/_logs/history/job_201301081658_0011_1357847072863_username_TableToFile%5F1357847071434 + -rw-r--r-- 1 username supergroup 26172 2013-01-10 14:44 /tmp/output/_logs/history/job_201301081658_0011_conf.xml + -rw-r--r-- 1 username supergroup 50 2013-01-10 14:44 /tmp/output/part-m-00000 + +We can see the output of our little map-reduce job: + + $ hadoop fs -text /tmp/output/output/part-m-00000 + catrow cf:cq [] catvalue + dogrow cf:cq [] dogvalue + $ + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/terasort.html ---------------------------------------------------------------------- diff --git a/1.8/examples/terasort.html b/1.8/examples/terasort.html index 178deff..b12b713 100644 --- a/1.8/examples/terasort.html +++ b/1.8/examples/terasort.html @@ -25,7 +25,7 @@ -Apache Accumulo Terasort Example +Apache Accumulo™ @@ -84,7 +84,7 @@
    • Releases
      • -

      Apache Accumulo Terasort Example

      +

      Apache Accumulo™

      -

      This example uses map/reduce to generate random input data that will +

      Title: Apache Accumulo Terasort Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      This example uses map/reduce to generate random input data that will be sorted by storing it into accumulo. It uses data very similar to the hadoop terasort benchmark.

      http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/terasort.md ---------------------------------------------------------------------- diff --git a/1.8/examples/terasort.md b/1.8/examples/terasort.md new file mode 100644 index 0000000..409c1d1 --- /dev/null +++ b/1.8/examples/terasort.md @@ -0,0 +1,50 @@ +Title: Apache Accumulo Terasort Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example uses map/reduce to generate random input data that will +be sorted by storing it into accumulo. It uses data very similar to the +hadoop terasort benchmark. + +To run this example you run it with arguments describing the amount of data: + + $ bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.TeraSortIngest \ + -i instance -z zookeepers -u user -p password \ + --count 10 \ + --minKeySize 10 \ + --maxKeySize 10 \ + --minValueSize 78 \ + --maxValueSize 78 \ + --table sort \ + --splits 10 \ + +After the map reduce job completes, scan the data: + + $ ./bin/accumulo shell -u username -p password + username@instance> scan -t sort + +l-$$OE/ZH c: 4 [] GGGGGGGGGGWWWWWWWWWWMMMMMMMMMMCCCCCCCCCCSSSSSSSSSSIIIIIIIIIIYYYYYYYYYYOOOOOOOO + ,C)wDw//u= c: 10 [] CCCCCCCCCCSSSSSSSSSSIIIIIIIIIIYYYYYYYYYYOOOOOOOOOOEEEEEEEEEEUUUUUUUUUUKKKKKKKK + 75@~?'WdUF c: 1 [] IIIIIIIIIIYYYYYYYYYYOOOOOOOOOOEEEEEEEEEEUUUUUUUUUUKKKKKKKKKKAAAAAAAAAAQQQQQQQQ + ;L+!2rT~hd c: 8 [] MMMMMMMMMMCCCCCCCCCCSSSSSSSSSSIIIIIIIIIIYYYYYYYYYYOOOOOOOOOOEEEEEEEEEEUUUUUUUU + LsS8)|.ZLD c: 5 [] OOOOOOOOOOEEEEEEEEEEUUUUUUUUUUKKKKKKKKKKAAAAAAAAAAQQQQQQQQQQGGGGGGGGGGWWWWWWWW + M^*dDE;6^< c: 9 [] UUUUUUUUUUKKKKKKKKKKAAAAAAAAAAQQQQQQQQQQGGGGGGGGGGWWWWWWWWWWMMMMMMMMMMCCCCCCCC + ^Eu) -Apache Accumulo Visibility, Authorizations, and Permissions Example +Creating a new user @@ -84,7 +84,7 @@
    • Releases
      • -

      Apache Accumulo Visibility, Authorizations, and Permissions Example

      +

      Creating a new user

      -

      Creating a new user

      +

      Title: Apache Accumulo Visibility, Authorizations, and Permissions Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + “License”); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License.

      + +

      Creating a new user

      root@instance> createuser username
 Enter new password for 'username': ********

http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/1.8/examples/visibility.md
----------------------------------------------------------------------
diff --git a/1.8/examples/visibility.md b/1.8/examples/visibility.md
new file mode 100644
index 0000000..b766dba
--- /dev/null
+++ b/1.8/examples/visibility.md
@@ -0,0 +1,131 @@
+Title: Apache Accumulo Visibility, Authorizations, and Permissions Example
+Notice:    Licensed to the Apache Software Foundation (ASF) under one
+           or more contributor license agreements.  See the NOTICE file
+           distributed with this work for additional information
+           regarding copyright ownership.  The ASF licenses this file
+           to you under the Apache License, Version 2.0 (the
+           "License"); you may not use this file except in compliance
+           with the License.  You may obtain a copy of the License at
+           .
+             http://www.apache.org/licenses/LICENSE-2.0
+           .
+           Unless required by applicable law or agreed to in writing,
+           software distributed under the License is distributed on an
+           "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+           KIND, either express or implied.  See the License for the
+           specific language governing permissions and limitations
+           under the License.
+
+## Creating a new user
+
+    root@instance> createuser username
+    Enter new password for 'username': ********
+    Please confirm new password for 'username': ********
+    root@instance> user username
+    Enter password for user username: ********
+    username@instance> createtable vistest
+    06 10:48:47,931 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloSecurityException: Error PERMISSION_DENIED - User does not have permission to perform this action
+    username@instance> userpermissions
+    System permissions:
+
+    Table permissions (accumulo.metadata): Table.READ
+    username@instance>
+
+A user does not by default have permission to create a table.
+
+## Granting permissions to a user
+
+    username@instance> user root
+    Enter password for user root: ********
+    root@instance> grant -s System.CREATE_TABLE -u username
+    root@instance> user username
+    Enter password for user username: ********
+    username@instance> createtable vistest
+    username@instance> userpermissions
+    System permissions: System.CREATE_TABLE
+
+    Table permissions (accumulo.metadata): Table.READ
+    Table permissions (vistest): Table.READ, Table.WRITE, Table.BULK_IMPORT, Table.ALTER_TABLE, Table.GRANT, Table.DROP_TABLE
+    username@instance vistest>
+
+## Inserting data with visibilities
+
+Visibilities are boolean AND (&) and OR (|) combinations of authorization
+tokens. Authorization tokens are arbitrary strings taken from a restricted
+ASCII character set. Parentheses are required to specify order of operations
+in visibilities.
+
+    username@instance vistest> insert row f1 q1 v1 -l A
+    username@instance vistest> insert row f2 q2 v2 -l A&B
+    username@instance vistest> insert row f3 q3 v3 -l apple&carrot|broccoli|spinach
+    06 11:19:01,432 [shell.Shell] ERROR: org.apache.accumulo.core.util.BadArgumentException: cannot mix | and & near index 12
+    apple&carrot|broccoli|spinach
+                ^
+    username@instance vistest> insert row f3 q3 v3 -l (apple&carrot)|broccoli|spinach
+    username@instance vistest>
+
+## Scanning with authorizations
+
+Authorizations are sets of authorization tokens. Each Accumulo user has
+authorizations and each Accumulo scan has authorizations. Scan authorizations
+are only allowed to be a subset of the user's authorizations. By default, a
+user's authorizations set is empty.
+
+    username@instance vistest> scan
+    username@instance vistest> scan -s A
+    06 11:43:14,951 [shell.Shell] ERROR: java.lang.RuntimeException: org.apache.accumulo.core.client.AccumuloSecurityException: Error BAD_AUTHORIZATIONS - The user does not have the specified authorizations assigned
+    username@instance vistest>
+
+## Setting authorizations for a user
+
+    username@instance vistest> setauths -s A
+    06 11:53:42,056 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloSecurityException: Error PERMISSION_DENIED - User does not have permission to perform this action
+    username@instance vistest>
+
+A user cannot set authorizations unless the user has the System.ALTER_USER permission.
+The root user has this permission.
+
+    username@instance vistest> user root
+    Enter password for user root: ********
+    root@instance vistest> setauths -s A -u username
+    root@instance vistest> user username
+    Enter password for user username: ********
+    username@instance vistest> scan -s A
+    row f1:q1 [A]    v1
+    username@instance vistest> scan
+    row f1:q1 [A]    v1
+    username@instance vistest>
+
+The default authorizations for a scan are the user's entire set of authorizations.
+
+    username@instance vistest> user root
+    Enter password for user root: ********
+    root@instance vistest> setauths -s A,B,broccoli -u username
+    root@instance vistest> user username
+    Enter password for user username: ********
+    username@instance vistest> scan
+    row f1:q1 [A]    v1
+    row f2:q2 [A&B]    v2
+    row f3:q3 [(apple&carrot)|broccoli|spinach]    v3
+    username@instance vistest> scan -s B
+    username@instance vistest>
+
+If you want, you can limit a user to only be able to insert data which they can read themselves.
+It can be set with the following constraint.
+
+    username@instance vistest> user root
+    Enter password for user root: ******
+    root@instance vistest> config -t vistest -s table.constraint.1=org.apache.accumulo.core.security.VisibilityConstraint
+    root@instance vistest> user username
+    Enter password for user username: ********
+    username@instance vistest> insert row f4 q4 v4 -l spinach
+        Constraint Failures:
+            ConstraintViolationSummary(constrainClass:org.apache.accumulo.core.security.VisibilityConstraint, violationCode:2, violationDescription:User does not have authorization on column visibility, numberOfViolatingMutations:1)
+    username@instance vistest> insert row f4 q4 v4 -l spinach|broccoli
+    username@instance vistest> scan
+    row f1:q1 [A]    v1
+    row f2:q2 [A&B]    v2
+    row f3:q3 [(apple&carrot)|broccoli|spinach]    v3
+    row f4:q4 [spinach|broccoli]    v4
+    username@instance vistest>
+

http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/blog/2014/05/03/accumulo-classloader.html
----------------------------------------------------------------------
diff --git a/blog/2014/05/03/accumulo-classloader.html b/blog/2014/05/03/accumulo-classloader.html
index 5432d8a..7ae3896 100644
--- a/blog/2014/05/03/accumulo-classloader.html
+++ b/blog/2014/05/03/accumulo-classloader.html
@@ -84,7 +84,7 @@
         
    • 
         Releases
         
        
-          
      • 1.8.0 (Latest)
        • 
+          
      • 1.8.1 (Latest)
        • 
           
      • 1.7.2
        • 
           
      • 1.6.6
        • 
           
      • Archive
        • 

http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/1d06e96a/blog/2014/05/27/getting-started-with-accumulo-1.6.0.html
----------------------------------------------------------------------
diff --git a/blog/2014/05/27/getting-started-with-accumulo-1.6.0.html b/blog/2014/05/27/getting-started-with-accumulo-1.6.0.html
index 039081e..d6f2296 100644
--- a/blog/2014/05/27/getting-started-with-accumulo-1.6.0.html
+++ b/blog/2014/05/27/getting-started-with-accumulo-1.6.0.html
@@ -84,7 +84,7 @@
         
      • 
         Releases