Return-Path: X-Original-To: apmail-accumulo-commits-archive@www.apache.org Delivered-To: apmail-accumulo-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 32C4B17543 for ; Mon, 18 May 2015 04:27:41 +0000 (UTC) Received: (qmail 91060 invoked by uid 500); 18 May 2015 04:27:41 -0000 Delivered-To: apmail-accumulo-commits-archive@accumulo.apache.org Received: (qmail 91021 invoked by uid 500); 18 May 2015 04:27:41 -0000 Mailing-List: contact commits-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list commits@accumulo.apache.org Received: (qmail 90926 invoked by uid 99); 18 May 2015 04:27:40 -0000 Received: from eris.apache.org (HELO hades.apache.org) (140.211.11.105) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2015 04:27:40 +0000 Received: from hades.apache.org (localhost [127.0.0.1]) by hades.apache.org (ASF Mail Server at hades.apache.org) with ESMTP id C64E9AC1152 for ; Mon, 18 May 2015 04:27:40 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1679931 [4/4] - in /accumulo/site/trunk/content/1.7: ./ examples/ Date: Mon, 18 May 2015 04:27:40 -0000 To: commits@accumulo.apache.org From: elserj@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20150518042740.C64E9AC1152@hades.apache.org> Added: accumulo/site/trunk/content/1.7/examples/reservations.mdtext URL: http://svn.apache.org/viewvc/accumulo/site/trunk/content/1.7/examples/reservations.mdtext?rev=1679931&view=auto ============================================================================== --- accumulo/site/trunk/content/1.7/examples/reservations.mdtext (added) +++ accumulo/site/trunk/content/1.7/examples/reservations.mdtext Mon May 18 04:27:39 2015 @@ -0,0 +1,66 @@ +Title: Apache Accumulo Isolation Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example shows running a simple reservation system implemented using +conditional mutations. This system guarantees that only one concurrent user can +reserve a resource. The example's reserve command allows multiple users to be +specified. When this is done, it creates a separate reservation thread for each +user. In the example below threads are spun up for alice, bob, eve, mallory, +and trent to reserve room06 on 20140101. Bob ends up getting the reservation +and everyone else is put on a wait list. The example code will take any string +for what, when and who. + + $ ./bin/accumulo org.apache.accumulo.examples.simple.reservations.ARS + >connect test16 localhost root secret ars + connected + > + Commands : + reserve {who} + cancel + list + >reserve room06 20140101 alice bob eve mallory trent + bob : RESERVED + mallory : WAIT_LISTED + alice : WAIT_LISTED + trent : WAIT_LISTED + eve : WAIT_LISTED + >list room06 20140101 + Reservation holder : bob + Wait list : [mallory, alice, trent, eve] + >cancel room06 20140101 alice + >cancel room06 20140101 bob + >list room06 20140101 + Reservation holder : mallory + Wait list : [trent, eve] + >quit + +Scanning the table in the Accumulo shell after running the example shows the +following: + + root@test16> table ars + root@test16 ars> scan + room06:20140101 res:0001 [] mallory + room06:20140101 res:0003 [] trent + room06:20140101 res:0004 [] eve + room06:20140101 tx:seq [] 6 + +The tx:seq column is incremented for each update to the row allowing for +detection of concurrent changes. For an update to go through, the sequence +number must not have changed since the data was read. If it does change, +the conditional mutation will fail and the example code will retry. + Added: accumulo/site/trunk/content/1.7/examples/rowhash.mdtext URL: http://svn.apache.org/viewvc/accumulo/site/trunk/content/1.7/examples/rowhash.mdtext?rev=1679931&view=auto ============================================================================== --- accumulo/site/trunk/content/1.7/examples/rowhash.mdtext (added) +++ accumulo/site/trunk/content/1.7/examples/rowhash.mdtext Mon May 18 04:27:39 2015 @@ -0,0 +1,59 @@ +Title: Apache Accumulo RowHash Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example shows a simple map/reduce job that reads from an accumulo table and +writes back into that table. + +To run this example you will need some data in a table. The following will +put a trivial amount of data into accumulo using the accumulo shell: + + $ ./bin/accumulo shell -u username -p password + Shell - Apache Accumulo Interactive Shell + - version: 1.6.0 + - instance name: instance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + username@instance> createtable input + username@instance> insert a-row cf cq value + username@instance> insert b-row cf cq value + username@instance> quit + +The RowHash class will insert a hash for each row in the database if it contains a +specified colum. Here's how you run the map/reduce job + + $ bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.RowHash -u user -p passwd -i instance -t input --column cf:cq + +Now we can scan the table and see the hashes: + + $ ./bin/accumulo shell -u username -p password + Shell - Apache Accumulo Interactive Shell + - version: 1.6.0 + - instance name: instance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + username@instance> scan -t input + a-row cf:cq [] value + a-row cf-HASHTYPE:cq-MD5BASE64 [] IGPBYI1uC6+AJJxC4r5YBA== + b-row cf:cq [] value + b-row cf-HASHTYPE:cq-MD5BASE64 [] IGPBYI1uC6+AJJxC4r5YBA== + username@instance> + Added: accumulo/site/trunk/content/1.7/examples/shard.mdtext URL: http://svn.apache.org/viewvc/accumulo/site/trunk/content/1.7/examples/shard.mdtext?rev=1679931&view=auto ============================================================================== --- accumulo/site/trunk/content/1.7/examples/shard.mdtext (added) +++ accumulo/site/trunk/content/1.7/examples/shard.mdtext Mon May 18 04:27:39 2015 @@ -0,0 +1,67 @@ +Title: Apache Accumulo Shard Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Accumulo has an iterator called the intersecting iterator which supports querying a term index that is partitioned by +document, or "sharded". This example shows how to use the intersecting iterator through these four programs: + + * Index.java - Indexes a set of text files into an Accumulo table + * Query.java - Finds documents containing a given set of terms. + * Reverse.java - Reads the index table and writes a map of documents to terms into another table. + * ContinuousQuery.java Uses the table populated by Reverse.java to select N random terms per document. Then it continuously and randomly queries those terms. + +To run these example programs, create two tables like below. + + username@instance> createtable shard + username@instance shard> createtable doc2term + +After creating the tables, index some files. The following command indexes all of the java files in the Accumulo source code. + + $ cd /local/username/workspace/accumulo/ + $ find core/src server/src -name "*.java" | xargs ./bin/accumulo org.apache.accumulo.examples.simple.shard.Index -i instance -z zookeepers -t shard -u username -p password --partitions 30 + +The following command queries the index to find all files containing 'foo' and 'bar'. + + $ cd $ACCUMULO_HOME + $ ./bin/accumulo org.apache.accumulo.examples.simple.shard.Query -i instance -z zookeepers -t shard -u username -p password foo bar + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/security/ColumnVisibilityTest.java + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/client/mock/MockConnectorTest.java + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/security/VisibilityEvaluatorTest.java + /local/username/workspace/accumulo/src/server/src/main/java/accumulo/test/functional/RowDeleteTest.java + /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/logger/TestLogWriter.java + /local/username/workspace/accumulo/src/server/src/main/java/accumulo/test/functional/DeleteEverythingTest.java + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/data/KeyExtentTest.java + /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/constraints/MetadataConstraintsTest.java + /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/iterators/WholeRowIteratorTest.java + /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/util/DefaultMapTest.java + /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/tabletserver/InMemoryMapTest.java + +In order to run ContinuousQuery, we need to run Reverse.java to populate doc2term. + + $ ./bin/accumulo org.apache.accumulo.examples.simple.shard.Reverse -i instance -z zookeepers --shardTable shard --doc2Term doc2term -u username -p password + +Below ContinuousQuery is run using 5 terms. So it selects 5 random terms from each document, then it continually +randomly selects one set of 5 terms and queries. It prints the number of matching documents and the time in seconds. + + $ ./bin/accumulo org.apache.accumulo.examples.simple.shard.ContinuousQuery -i instance -z zookeepers --shardTable shard --doc2Term doc2term -u username -p password --terms 5 + [public, core, class, binarycomparable, b] 2 0.081 + [wordtodelete, unindexdocument, doctablename, putdelete, insert] 1 0.041 + [import, columnvisibilityinterpreterfactory, illegalstateexception, cv, columnvisibility] 1 0.049 + [getpackage, testversion, util, version, 55] 1 0.048 + [for, static, println, public, the] 55 0.211 + [sleeptime, wrappingiterator, options, long, utilwaitthread] 1 0.057 + [string, public, long, 0, wait] 12 0.132 Added: accumulo/site/trunk/content/1.7/examples/tabletofile.mdtext URL: http://svn.apache.org/viewvc/accumulo/site/trunk/content/1.7/examples/tabletofile.mdtext?rev=1679931&view=auto ============================================================================== --- accumulo/site/trunk/content/1.7/examples/tabletofile.mdtext (added) +++ accumulo/site/trunk/content/1.7/examples/tabletofile.mdtext Mon May 18 04:27:39 2015 @@ -0,0 +1,59 @@ +Title: Apache Accumulo Table-to-File Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example uses mapreduce to extract specified columns from an existing table. + +To run this example you will need some data in a table. The following will +put a trivial amount of data into accumulo using the accumulo shell: + + $ ./bin/accumulo shell -u username -p password + Shell - Apache Accumulo Interactive Shell + - version: 1.6.0 + - instance name: instance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + username@instance> createtable input + username@instance> insert dog cf cq dogvalue + username@instance> insert cat cf cq catvalue + username@instance> insert junk family qualifier junkvalue + username@instance> quit + +The TableToFile class configures a map-only job to read the specified columns and +write the key/value pairs to a file in HDFS. + +The following will extract the rows containing the column "cf:cq": + + $ bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.TableToFile -u user -p passwd -i instance -t input --columns cf:cq --output /tmp/output + + $ hadoop fs -ls /tmp/output + -rw-r--r-- 1 username supergroup 0 2013-01-10 14:44 /tmp/output/_SUCCESS + drwxr-xr-x - username supergroup 0 2013-01-10 14:44 /tmp/output/_logs + drwxr-xr-x - username supergroup 0 2013-01-10 14:44 /tmp/output/_logs/history + -rw-r--r-- 1 username supergroup 9049 2013-01-10 14:44 /tmp/output/_logs/history/job_201301081658_0011_1357847072863_username_TableToFile%5F1357847071434 + -rw-r--r-- 1 username supergroup 26172 2013-01-10 14:44 /tmp/output/_logs/history/job_201301081658_0011_conf.xml + -rw-r--r-- 1 username supergroup 50 2013-01-10 14:44 /tmp/output/part-m-00000 + +We can see the output of our little map-reduce job: + + $ hadoop fs -text /tmp/output/output/part-m-00000 + catrow cf:cq [] catvalue + dogrow cf:cq [] dogvalue + $ + Added: accumulo/site/trunk/content/1.7/examples/terasort.mdtext URL: http://svn.apache.org/viewvc/accumulo/site/trunk/content/1.7/examples/terasort.mdtext?rev=1679931&view=auto ============================================================================== --- accumulo/site/trunk/content/1.7/examples/terasort.mdtext (added) +++ accumulo/site/trunk/content/1.7/examples/terasort.mdtext Mon May 18 04:27:39 2015 @@ -0,0 +1,50 @@ +Title: Apache Accumulo Terasort Example +Notice: Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +This example uses map/reduce to generate random input data that will +be sorted by storing it into accumulo. It uses data very similar to the +hadoop terasort benchmark. + +To run this example you run it with arguments describing the amount of data: + + $ bin/tool.sh lib/accumulo-examples-simple.jar org.apache.accumulo.examples.simple.mapreduce.TeraSortIngest \ + -i instance -z zookeepers -u user -p password \ + --count 10 \ + --minKeySize 10 \ + --maxKeySize 10 \ + --minValueSize 78 \ + --maxValueSize 78 \ + --table sort \ + --splits 10 \ + +After the map reduce job completes, scan the data: + + $ ./bin/accumulo shell -u username -p password + username@instance> scan -t sort + +l-$$OE/ZH c: 4 [] GGGGGGGGGGWWWWWWWWWWMMMMMMMMMMCCCCCCCCCCSSSSSSSSSSIIIIIIIIIIYYYYYYYYYYOOOOOOOO + ,C)wDw//u= c: 10 [] CCCCCCCCCCSSSSSSSSSSIIIIIIIIIIYYYYYYYYYYOOOOOOOOOOEEEEEEEEEEUUUUUUUUUUKKKKKKKK + 75@~?'WdUF c: 1 [] IIIIIIIIIIYYYYYYYYYYOOOOOOOOOOEEEEEEEEEEUUUUUUUUUUKKKKKKKKKKAAAAAAAAAAQQQQQQQQ + ;L+!2rT~hd c: 8 [] MMMMMMMMMMCCCCCCCCCCSSSSSSSSSSIIIIIIIIIIYYYYYYYYYYOOOOOOOOOOEEEEEEEEEEUUUUUUUU + LsS8)|.ZLD c: 5 [] OOOOOOOOOOEEEEEEEEEEUUUUUUUUUUKKKKKKKKKKAAAAAAAAAAQQQQQQQQQQGGGGGGGGGGWWWWWWWW + M^*dDE;6^< c: 9 [] UUUUUUUUUUKKKKKKKKKKAAAAAAAAAAQQQQQQQQQQGGGGGGGGGGWWWWWWWWWWMMMMMMMMMMCCCCCCCC + ^Eu) createuser username + Enter new password for 'username': ******** + Please confirm new password for 'username': ******** + root@instance> user username + Enter password for user username: ******** + username@instance> createtable vistest + 06 10:48:47,931 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloSecurityException: Error PERMISSION_DENIED - User does not have permission to perform this action + username@instance> userpermissions + System permissions: + + Table permissions (accumulo.metadata): Table.READ + username@instance> + +A user does not by default have permission to create a table. + +## Granting permissions to a user + + username@instance> user root + Enter password for user root: ******** + root@instance> grant -s System.CREATE_TABLE -u username + root@instance> user username + Enter password for user username: ******** + username@instance> createtable vistest + username@instance> userpermissions + System permissions: System.CREATE_TABLE + + Table permissions (accumulo.metadata): Table.READ + Table permissions (vistest): Table.READ, Table.WRITE, Table.BULK_IMPORT, Table.ALTER_TABLE, Table.GRANT, Table.DROP_TABLE + username@instance vistest> + +## Inserting data with visibilities + +Visibilities are boolean AND (&) and OR (|) combinations of authorization +tokens. Authorization tokens are arbitrary strings taken from a restricted +ASCII character set. Parentheses are required to specify order of operations +in visibilities. + + username@instance vistest> insert row f1 q1 v1 -l A + username@instance vistest> insert row f2 q2 v2 -l A&B + username@instance vistest> insert row f3 q3 v3 -l apple&carrot|broccoli|spinach + 06 11:19:01,432 [shell.Shell] ERROR: org.apache.accumulo.core.util.BadArgumentException: cannot mix | and & near index 12 + apple&carrot|broccoli|spinach + ^ + username@instance vistest> insert row f3 q3 v3 -l (apple&carrot)|broccoli|spinach + username@instance vistest> + +## Scanning with authorizations + +Authorizations are sets of authorization tokens. Each Accumulo user has +authorizations and each Accumulo scan has authorizations. Scan authorizations +are only allowed to be a subset of the user's authorizations. By default, a +user's authorizations set is empty. + + username@instance vistest> scan + username@instance vistest> scan -s A + 06 11:43:14,951 [shell.Shell] ERROR: java.lang.RuntimeException: org.apache.accumulo.core.client.AccumuloSecurityException: Error BAD_AUTHORIZATIONS - The user does not have the specified authorizations assigned + username@instance vistest> + +## Setting authorizations for a user + + username@instance vistest> setauths -s A + 06 11:53:42,056 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloSecurityException: Error PERMISSION_DENIED - User does not have permission to perform this action + username@instance vistest> + +A user cannot set authorizations unless the user has the System.ALTER_USER permission. +The root user has this permission. + + username@instance vistest> user root + Enter password for user root: ******** + root@instance vistest> setauths -s A -u username + root@instance vistest> user username + Enter password for user username: ******** + username@instance vistest> scan -s A + row f1:q1 [A] v1 + username@instance vistest> scan + row f1:q1 [A] v1 + username@instance vistest> + +The default authorizations for a scan are the user's entire set of authorizations. + + username@instance vistest> user root + Enter password for user root: ******** + root@instance vistest> setauths -s A,B,broccoli -u username + root@instance vistest> user username + Enter password for user username: ******** + username@instance vistest> scan + row f1:q1 [A] v1 + row f2:q2 [A&B] v2 + row f3:q3 [(apple&carrot)|broccoli|spinach] v3 + username@instance vistest> scan -s B + username@instance vistest> + +If you want, you can limit a user to only be able to insert data which they can read themselves. +It can be set with the following constraint. + + username@instance vistest> user root + Enter password for user root: ****** + root@instance vistest> config -t vistest -s table.constraint.1=org.apache.accumulo.core.security.VisibilityConstraint + root@instance vistest> user username + Enter password for user username: ******** + username@instance vistest> insert row f4 q4 v4 -l spinach + Constraint Failures: + ConstraintViolationSummary(constrainClass:org.apache.accumulo.core.security.VisibilityConstraint, violationCode:2, violationDescription:User does not have authorization on column visibility, numberOfViolatingMutations:1) + username@instance vistest> insert row f4 q4 v4 -l spinach|broccoli + username@instance vistest> scan + row f1:q1 [A] v1 + row f2:q2 [A&B] v2 + row f3:q3 [(apple&carrot)|broccoli|spinach] v3 + row f4:q4 [spinach|broccoli] v4 + username@instance vistest> +