accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From els...@apache.org
Subject [3/3] git commit: Merge branch '1.6.1-SNAPSHOT'
Date Thu, 07 Aug 2014 18:32:44 GMT
Merge branch '1.6.1-SNAPSHOT'

Conflicts:
	docs/src/main/asciidoc/chapters/administration.txt
	docs/src/main/asciidoc/chapters/clients.txt


Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/c3907969
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/c3907969
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/c3907969

Branch: refs/heads/master
Commit: c390796911f9f47a787966eb6f3702aabddb9bf3
Parents: d440049 d5e094d
Author: Josh Elser <elserj@apache.org>
Authored: Thu Aug 7 14:32:18 2014 -0400
Committer: Josh Elser <elserj@apache.org>
Committed: Thu Aug 7 14:32:18 2014 -0400

----------------------------------------------------------------------
 docs/src/main/asciidoc/chapters/administration.txt |  4 ++++
 docs/src/main/asciidoc/chapters/clients.txt        | 10 ++++++++++
 2 files changed, 14 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/accumulo/blob/c3907969/docs/src/main/asciidoc/chapters/administration.txt
----------------------------------------------------------------------
diff --cc docs/src/main/asciidoc/chapters/administration.txt
index 5e92465,0000000..9817b07
mode 100644,000000..100644
--- a/docs/src/main/asciidoc/chapters/administration.txt
+++ b/docs/src/main/asciidoc/chapters/administration.txt
@@@ -1,492 -1,0 +1,496 @@@
 +// Licensed to the Apache Software Foundation (ASF) under one or more
 +// contributor license agreements.  See the NOTICE file distributed with
 +// this work for additional information regarding copyright ownership.
 +// The ASF licenses this file to You under the Apache License, Version 2.0
 +// (the "License"); you may not use this file except in compliance with
 +// the License.  You may obtain a copy of the License at
 +//
 +//     http://www.apache.org/licenses/LICENSE-2.0
 +//
 +// Unless required by applicable law or agreed to in writing, software
 +// distributed under the License is distributed on an "AS IS" BASIS,
 +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 +// See the License for the specific language governing permissions and
 +// limitations under the License.
 +
 +== Administration
 +
 +=== Hardware
 +
 +Because we are running essentially two or three systems simultaneously layered
 +across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
 +consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
 +at least one core and 2 - 4 GB each.
 +
 +One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
 +typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.
 +
 +It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
 +each, but in this case it is recommended to only run up to two processes per
 +machine -- i.e. DataNode and TabletServer or DataNode and MapReduce worker but
 +not all three. The constraint here is having enough available heap space for all the
 +processes on a machine.
 +
 +=== Network
 +
 +Accumulo communicates via remote procedure calls over TCP/IP for both passing
 +data and control messages. In addition, Accumulo uses HDFS clients to
 +communicate with HDFS. To achieve good ingest and query performance, sufficient
 +network bandwidth must be available between any two machines.
 +
 +In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
 +use the following default ports. Please make sure that they are open, or change
 +their value in conf/accumulo-site.xml.
 +
 +.Accumulo default ports
 +[width="75%",cols=">,^2,^2"]
 +[options="header"]
 +|====
 +|Port | Description | Property Name
 +|4445 | Shutdown Port (Accumulo MiniCluster) | n/a
 +|4560 | Accumulo monitor (for centralized log display) | monitor.port.log4j
 +|9997 | Tablet Server | tserver.port.client
 +|9999 | Master Server | master.port.client
 +|12234 | Accumulo Tracer | trace.port.client
 +|42424 | Accumulo Proxy Server | n/a
 +|50091 | Accumulo GC | gc.port.client
 +|50095 | Accumulo HTTP monitor | monitor.port.client
 +|====
 +
 +In addition, the user can provide +0+ and an ephemeral port will be chosen instead. This
 +ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
 +use +0+ instead of an explicit value, should, in most cases, work around any issues of
 +running multiple distinct Accumulo instances (or any other process which tries to use the
 +same default ports) on the same hardware.
 +
 +=== Installation
 +Choose a directory for the Accumulo installation. This directory will be referenced
 +by the environment variable +$ACCUMULO_HOME+. Run the following:
 +
 +  $ tar xzf accumulo-1.6.0-bin.tar.gz    # unpack to subdirectory
 +  $ mv accumulo-1.6.0 $ACCUMULO_HOME # move to desired location
 +
 +Repeat this step at each machine within the cluster. Usually all machines have the
 +same +$ACCUMULO_HOME+.
 +
 +=== Dependencies
 +Accumulo requires HDFS and ZooKeeper to be configured and running
 +before starting. Password-less SSH should be configured between at least the
 +Accumulo master and TabletServer machines. It is also a good idea to run Network
 +Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of
 +sync, which can cause problems with automatically timestamped data.
 +
 +=== Configuration
 +
 +Accumulo is configured by editing several Shell and XML files found in
 ++$ACCUMULO_HOME/conf+. The structure closely resembles Hadoop's configuration
 +files.
 +
 +==== Edit conf/accumulo-env.sh
 +
 +Accumulo needs to know where to find the software it depends on. Edit accumulo-env.sh
 +and specify the following:
 +
 +. Enter the location of the installation directory of Accumulo for +$ACCUMULO_HOME+
 +. Enter your system's Java home for +$JAVA_HOME+
 +. Enter the location of Hadoop for +$HADOOP_PREFIX+
 +. Choose a location for Accumulo logs and enter it for +$ACCUMULO_LOG_DIR+
 +. Enter the location of ZooKeeper for +$ZOOKEEPER_HOME+
 +
 +By default Accumulo TabletServers are set to use 1GB of memory. You may change
 +this by altering the value of +$ACCUMULO_TSERVER_OPTS+. Note the syntax is that of
 +the Java JVM command line options. This value should be less than the physical
 +memory of the machines running TabletServers.
 +
 +There are similar options for the master's memory usage and the garbage collector
 +process. Reduce these if they exceed the physical RAM of your hardware and
 +increase them, within the bounds of the physical RAM, if a process fails because of
 +insufficient memory.
 +
 +Note that you will be specifying the Java heap space in accumulo-env.sh. You should
 +make sure that the total heap space used for the Accumulo tserver and the Hadoop
 +DataNode and TaskTracker is less than the available memory on each slave node in
 +the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop
 +NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate
 +machines to allow them to use more heap space. If you are running these on the
 +same machine on a small cluster, likewise make sure their heap space settings fit
 +within the available memory.
 +
 +==== Native Map
 +
 +The tablet server uses a data structure called a MemTable to store sorted key/value
 +pairs in memory when they are first received from the client. When a minor compaction
 +occurs, this data structure is written to HDFS. The MemTable will default to using
 +memory in the JVM but a JNI version, called the native map, can be used to significantly
 +speed up performance by utilizing the memory space of the native operating system. The
 +native map also avoids the performance implications brought on by garbage collection
 +in the JVM by causing it to pause much less frequently.
 +
 +32-bit and 64-bit Linux and Mac OS X versions of the native map can be built
 +from the Accumulo bin package by executing
 ++$ACCUMULO_HOME/bin/build_native_library.sh+. If your system's
 +default compiler options are insufficient, you can add additional compiler
 +options to the command line, such as options for the architecture. These will be
 +passed to the Makefile in the environment variable +USERFLAGS+.
 +
 +Examples:
 +
 +. +$ACCUMULO_HOME/bin/build_native_library.sh+
 +. +$ACCUMULO_HOME/bin/build_native_library.sh -m32+
 +
 +After building the native map from the source, you will find the artifact in
 ++$ACCUMULO_HOME/lib/native+. Upon starting up, the tablet server will look
 +in this directory for the map library. If the file is renamed or moved from its
 +target directory, the tablet server may not be able to find it. The system can
 +also locate the native maps shared library by setting +LD_LIBRARY_PATH+
 +(or +DYLD_LIBRARY_PATH+ on Mac OS X) in +$ACCUMULO_HOME/conf/accumulo-env.sh+.
 +
 +==== Cluster Specification
 +
 +On the machine that will serve as the Accumulo master:
 +
 +. Write the IP address or domain name of the Accumulo Master to the +$ACCUMULO_HOME/conf/masters+
file.
 +. Write the IP addresses or domain name of the machines that will be TabletServers in +$ACCUMULO_HOME/conf/slaves+,
one per line.
 +
 +Note that if using domain names rather than IP addresses, DNS must be configured
 +properly for all machines participating in the cluster. DNS can be a confusing source
 +of errors.
 +
 +==== Accumulo Settings
 +Specify appropriate values for the following settings in
 ++$ACCUMULO_HOME/conf/accumulo-site.xml+ :
 +
 +[source,xml]
 +<property>
 +    <name>instance.zookeeper.host</name>
 +    <value>zooserver-one:2181,zooserver-two:2181</value>
 +    <description>list of zookeeper servers</description>
 +</property>
 +
 +This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate
 +settings between processes and helps finalize TabletServer failure.
 +
 +[source,xml]
 +<property>
 +    <name>instance.secret</name>
 +    <value>DEFAULT</value>
 +</property>
 +
 +The instance needs a secret to enable secure communication between servers. Configure your
 +secret and make sure that the +accumulo-site.xml+ file is not readable to other users.
 +For alternatives to storing the +instance.secret+ in plaintext, please read the
 ++Sensitive Configuration Values+ section.
 +
 +Some settings can be modified via the Accumulo shell and take effect immediately, but
 +some settings require a process restart to take effect. See the configuration documentation
 +(available in the docs directory of the tarball and in <<configuration>>) for
details.
 +
 +==== Deploy Configuration
 +
 +Copy the masters, slaves, accumulo-env.sh, and if necessary, accumulo-site.xml
 +from the +$ACCUMULO_HOME/conf/+ directory on the master to all the machines
 +specified in the slaves file.
 +
 +==== Sensitive Configuration Values
 +
 +Accumulo has a number of properties that can be specified via the accumulo-site.xml
 +file which are sensitive in nature, instance.secret and trace.token.property.password
 +are two common examples. Both of these properties, if compromised, have the ability
 +to result in data being leaked to users who should not have access to that data.
 +
 +In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves as a common
 +implementation to abstract away the storage and retrieval of passwords from plaintext
 +storage in configuration files. Any Property marked with the +Sensitive+ annotation
 +is a candidate for use with these CredentialProviders. For version of Hadoop which lack
 +these classes, the feature will just be unavailable for use.
 +
 +A comma separated list of CredentialProviders can be configured using the Accumulo Property
 ++general.security.credential.provider.paths+. Each configured URL will be consulted
 +when the Configuration object for accumulo-site.xml is accessed.
 +
 +==== Using a JavaKeyStoreCredentialProvider for storage
 +
 +One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore CredentialProvider.
 +Each entry in the KeyStore is the Accumulo Property key name. For example, to store the
 +\texttt{instance.secret}, the following command can be used:
 +
 +  hadoop credential create instance.secret --provider jceks://file/etc/accumulo/conf/accumulo.jceks
 +
 +The command will then prompt you to enter the secret to use and create a keystore in: 
 +
 +  /etc/accumulo/conf/accumulo.jceks
 +
 +Then, accumulo-site.xml must be configured to use this KeyStore as a CredentialProvider:
 +
 +[source,xml]
 +<property>
 +    <name>general.security.credential.provider.paths</name>
 +    <value>jceks://file/etc/accumulo/conf/accumulo.jceks</value>
 +</property>
 +
 +This configuration will then transparently extract the +instance.secret+ from
 +the configured KeyStore and alleviates a human readable storage of the sensitive
 +property.
 +
++A KeyStore can also be stored in HDFS, which will make the KeyStore readily available to
++all Accumulo servers. If the local filesystem is used, be aware that each Accumulo server
++will expect the KeyStore in the same location.
++
 +=== Initialization
 +
 +Accumulo must be initialized to create the structures it uses internally to locate
 +data across the cluster. HDFS is required to be configured and running before
 +Accumulo can be initialized.
 +
 +Once HDFS is started, initialization can be performed by executing
 ++$ACCUMULO_HOME/bin/accumulo init+ . This script will prompt for a name
 +for this instance of Accumulo. The instance name is used to identify a set of tables
 +and instance-specific settings. The script will then write some information into
 +HDFS so Accumulo can start properly.
 +
 +The initialization script will prompt you to set a root password. Once Accumulo is
 +initialized it can be started.
 +
 +=== Running
 +
 +==== Starting Accumulo
 +
 +Make sure Hadoop is configured on all of the machines in the cluster, including
 +access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running.
 +Make sure ZooKeeper is configured and running on at least one machine in the
 +cluster.
 +Start Accumulo using the +bin/start-all.sh+ script.
 +
 +To verify that Accumulo is running, check the Status page as described in
 +<<monitoring>>. In addition, the Shell can provide some information about the
status of
 +tables via reading the metadata tables.
 +
 +==== Stopping Accumulo
 +
 +To shutdown cleanly, run +bin/stop-all.sh+ and the master will orchestrate the
 +shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish,
so it may
 +take some time for particular configurations.
 +
 +==== Adding a Node
 +
 +Update your +$ACCUMULO_HOME/conf/slaves+ (or +$ACCUMULO_CONF_DIR/slaves+) file to account
for the addition.
 +
 +  $ACCUMULO_HOME/bin/accumulo admin start <host(s)> {<host> ...}
 +
 +Alternatively, you can ssh to each of the hosts you want to add and run:
 +
 +  $ACCUMULO_HOME/bin/start-here.sh
 +
 +Make sure the host in question has the new configuration, or else the tablet
 +server won't start; at a minimum this needs to be on the host(s) being added,
 +but in practice it's good to ensure consistent configuration across all nodes.
 +
 +==== Decomissioning a Node
 +
 +If you need to take a node out of operation, you can trigger a graceful shutdown of a tablet
 +server. Accumulo will automatically rebalance the tablets across the available tablet servers.
 +
 +  $ACCUMULO_HOME/bin/accumulo admin stop <host(s)> {<host> ...}
 +
 +Alternatively, you can ssh to each of the hosts you want to remove and run:
 +
 +  $ACCUMULO_HOME/bin/stop-here.sh
 +
 +Be sure to update your +$ACCUMULO_HOME/conf/slaves+ (or +$ACCUMULO_CONF_DIR/slaves+) file
to
 +account for the removal of these hosts. Bear in mind that the monitor will not re-read the
 +slaves file automatically, so it will report the decomissioned servers as down; it's
 +recommended that you restart the monitor so that the node list is up to date.
 +
 +[[monitoring]]
 +=== Monitoring
 +
 +The Accumulo Master provides an interface for monitoring the status and health of
 +Accumulo components. The Accumulo Monitor provides a web UI for accessing this information
at
 ++http://_monitorhost_:50095/+.
 +
 +Things highlighted in yellow may be in need of attention.
 +If anything is highlighted in red on the monitor page, it is something that definitely needs
attention.
 +
 +The Overview page contains some summary information about the Accumulo instance, including
the version, instance name, and instance ID.
 +There is a table labeled Accumulo Master with current status, a table listing the active
Zookeeper servers, and graphs displaying various metrics over time.
 +These include ingest and scan performance and other useful measurements.
 +
 +The Master Server, Tablet Servers, and Tables pages display metrics grouped in different
ways (e.g. by tablet server or by table).
 +Metrics typically include number of entries (key/value pairs), ingest and query rates.
 +The number of running scans, major and minor compactions are in the form _number_running_
(_number_queued_).
 +Another important metric is hold time, which is the amount of time a tablet has been waiting
but unable to flush its memory in a minor compaction.
 +
 +The Server Activity page graphically displays tablet server status, with each server represented
as a circle or square.
 +Different metrics may be assigned to the nodes' color and speed of oscillation.
 +The Overall Avg metric is only used on the Server Activity page, and represents the average
of all the other metrics (after normalization).
 +Similarly, the Overall Max metric picks the metric with the maximum normalized value.
 +
 +The Garbage Collector page displays a list of garbage collection cycles, the number of files
found of each type (including deletion candidates in use and files actually deleted), and
the length of the deletion cycle.
 +The Traces page displays data for recent traces performed (see the following section for
information on <<tracing>>).
 +The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo
processes.
 +Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.
 +
 +==== SSL
 +SSL may be enabled for the monitor page by setting the following properties in the +accumulo-site.xml+
file:
 +
 +  monitor.ssl.keyStore
 +  monitor.ssl.keyStorePassword
 +  monitor.ssl.trustStore
 +  monitor.ssl.trustStorePassword
 +
 +If the Accumulo conf directory has been configured (in particular the +accumulo-env.sh+
file must be set up), the +generate_monitor_certificate.sh+ script in the Accumulo +bin+ directory
can be used to create the keystore and truststore files with random passwords.
 +The script will print out the properties that need to be added to the +accumulo-site.xml+
file.
 +The stores can also be generated manually with the Java +keytool+ command, whose usage can
be seen in the +generate_monitor_certificate.sh+ script.
 +
 +If desired, the SSL ciphers allowed for connections can be controlled via the following
properties in +accumulo-site.xml+:
 +
 +  monitor.ssl.include.ciphers
 +  monitor.ssl.exclude.ciphers
 +
 +If SSL is enabled, the monitor URL can only be accessed via https.
 +This also allows you to access the Accumulo shell through the monitor page.
 +The left navigation bar will have a new link to Shell.
 +An Accumulo user name and password must be entered for access to the shell.
 +
 +[[tracing]]
 +=== Tracing
 +It can be difficult to determine why some operations are taking longer
 +than expected. For example, you may be looking up items with very low
 +latency, but sometimes the lookups take much longer. Determining the
 +cause of the delay is difficult because the system is distributed, and
 +the typical lookup is fast.
 +
 +Accumulo has been instrumented to record the time that various
 +operations take when tracing is turned on. The fact that tracing is
 +enabled follows all the requests made on behalf of the user throughout
 +the distributed infrastructure of accumulo, and across all threads of
 +execution.
 +
 +These time spans will be inserted into the +trace+ table in
 +Accumulo. You can browse recent traces from the Accumulo monitor
 +page. You can also read the +trace+ table directly like any
 +other table.
 +
 +The design of Accumulo's distributed tracing follows that of
 +http://research.google.com/pubs/pub36356.html[Google's Dapper].
 +
 +==== Tracers
 +To collect traces, Accumulo needs at least one server listed in
 + +$ACCUMULO_HOME/conf/tracers+. The server collects traces
 +from clients and writes them to the +trace+ table. The Accumulo
 +user that the tracer connects to Accumulo with can be configured with
 +the following properties
 +
 +  trace.user
 +  trace.token.property.password
 +
 +==== Instrumenting a Client
 +Tracing can be used to measure a client operation, such as a scan, as
 +the operation traverses the distributed system. To enable tracing for
 +your application call
 +
 +[source,java]
 +DistributedTrace.enable(instance, new ZooReader(instance), hostname, "myApplication");
 +
 +Once tracing has been enabled, a client can wrap an operation in a trace.
 +
 +[source,java]
 +Trace.on("Client Scan");
 +BatchScanner scanner = conn.createBatchScanner(...);
 +// Configure your scanner
 +for (Entry entry : scanner) {
 +}
 +Trace.off();
 +
 +Additionally, the user can create additional Spans within a Trace.
 +
 +[source,java]
 +Trace.on("Client Update");
 +...
 +Span readSpan = Trace.start("Read");
 +...
 +readSpan.stop();
 +...
 +Span writeSpan = Trace.start("Write");
 +...
 +writeSpan.stop();
 +Trace.off();
 +
 +Like Dapper, Accumulo tracing supports user defined annotations to associate additional
data with a Trace.
 +
 +[source,java]
 +...
 +int numberOfEntriesRead = 0;
 +Span readSpan = Trace.start("Read");
 +// Do the read, update the counter
 +...
 +readSpan.data("Number of Entries Read", String.valueOf(numberOfEntriesRead));
 +
 +Some client operations may have a high volume within your
 +application. As such, you may wish to only sample a percentage of
 +operations for tracing. As seen below, the CountSampler can be used to
 +help enable tracing for 1-in-1000 operations
 +
 +[source,java]
 +Sampler sampler = new CountSampler(1000);
 +...
 +if (sampler.next()) {
 +  Trace.on("Read");
 +}
 +...
 +Trace.offNoFlush();
 +
 +It should be noted that it is safe to turn off tracing even if it
 +isn't currently active. The +Trace.offNoFlush()+ should be used if the
 +user does not wish to have +Trace.off()+ block while flushing trace
 +data.
 +
 +==== Viewing Collected Traces
 +To view collected traces, use the "Recent Traces" link on the Monitor
 +UI. You can also programmatically access and print traces using the
 ++TraceDump+ class.
 +
 +==== Tracing from the Shell
 +You can enable tracing for operations run from the shell by using the
 ++trace on+ and +trace off+ commands.
 +
 +----
 +root@test test> trace on
 +
 +root@test test> scan
 +a b:c []    d
 +
 +root@test test> trace off
 +Waiting for trace information
 +Waiting for trace information
 +Trace started at 2013/08/26 13:24:08.332
 +Time  Start  Service@Location       Name
 + 3628+0      shell@localhost shell:root
 +    8+1690     shell@localhost scan
 +    7+1691       shell@localhost scan:location
 +    6+1692         tserver@localhost startScan
 +    5+1692           tserver@localhost tablet read ahead 6
 +----
 +
 +=== Logging
 +Accumulo processes each write to a set of log files. By default these are found under
 ++$ACCUMULO/logs/+.
 +
 +=== Recovery
 +
 +In the event of TabletServer failure or error on shutting Accumulo down, some
 +mutations may not have been minor compacted to HDFS properly. In this case,
 +Accumulo will automatically reapply such mutations from the write-ahead log
 +either when the tablets from the failed server are reassigned by the Master (in the
 +case of a single TabletServer failure) or the next time Accumulo starts (in the event of
 +failure during shutdown).
 +
 +Recovery is performed by asking a tablet server to sort the logs so that tablets can easily
find their missing
 +updates. The sort status of each file is displayed on
 +Accumulo monitor status page. Once the recovery is complete any
 +tablets involved should return to an ``online'' state. Until then those tablets will be
 +unavailable to clients.
 +
 +The Accumulo client library is configured to retry failed mutations and in many
 +cases clients will be able to continue processing after the recovery process without
 +throwing an exception.

http://git-wip-us.apache.org/repos/asf/accumulo/blob/c3907969/docs/src/main/asciidoc/chapters/clients.txt
----------------------------------------------------------------------
diff --cc docs/src/main/asciidoc/chapters/clients.txt
index 6b071ba,0000000..48123a3
mode 100644,000000..100644
--- a/docs/src/main/asciidoc/chapters/clients.txt
+++ b/docs/src/main/asciidoc/chapters/clients.txt
@@@ -1,320 -1,0 +1,330 @@@
 +// Licensed to the Apache Software Foundation (ASF) under one or more
 +// contributor license agreements.  See the NOTICE file distributed with
 +// this work for additional information regarding copyright ownership.
 +// The ASF licenses this file to You under the Apache License, Version 2.0
 +// (the "License"); you may not use this file except in compliance with
 +// the License.  You may obtain a copy of the License at
 +//
 +//     http://www.apache.org/licenses/LICENSE-2.0
 +//
 +// Unless required by applicable law or agreed to in writing, software
 +// distributed under the License is distributed on an "AS IS" BASIS,
 +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 +// See the License for the specific language governing permissions and
 +// limitations under the License.
 +
 +== Writing Accumulo Clients
 +
 +=== Running Client Code
 +
 +There are multiple ways to run Java code that uses Accumulo. Below is a list
 +of the different ways to execute client code.
 +
 +* using java executable
 +* using the accumulo script
 +* using the tool script
 +
 +In order to run client code written to run against Accumulo, you will need to
 +include the jars that Accumulo depends on in your classpath. Accumulo client
 +code depends on Hadoop and Zookeeper. For Hadoop add the hadoop client jar, all
 +of the jars in the Hadoop lib directory, and the conf directory to the
 +classpath. For Zookeeper 3.3 you only need to add the Zookeeper jar, and not
 +what is in the Zookeeper lib directory. You can run the following command on a
 +configured Accumulo system to see what its using for its classpath.
 +
 +  $ACCUMULO_HOME/bin/accumulo classpath
 +
 +Another option for running your code is to put a jar file in
 ++$ACCUMULO_HOME/lib/ext+. After doing this you can use the accumulo
 +script to execute your code. For example if you create a jar containing the
 +class +com.foo.Client+ and placed that in +lib/ext+, then you could use the command
 ++$ACCUMULO_HOME/bin/accumulo com.foo.Client+ to execute your code.
 +
 +If you are writing map reduce job that access Accumulo, then you can use the
 +bin/tool.sh script to run those jobs. See the map reduce example.
 +
 +=== Connecting
 +
 +All clients must first identify the Accumulo instance to which they will be
 +communicating. Code to do this is as follows:
 +
 +[source,java]
 +----
 +String instanceName = "myinstance";
 +String zooServers = "zooserver-one,zooserver-two"
 +Instance inst = new ZooKeeperInstance(instanceName, zooServers);
 +
 +Connector conn = inst.getConnector("user", new PasswordToken("passwd"));
 +----
 +
++The PasswordToken is the most common implementation of an \texttt{AuthenticationToken}.
++This general interface allow authentication as an Accumulo user to come from
++a variety of sources or means. The CredentialProviderToken leverages the Hadoop
++CredentialProviders (new in Hadoop 2.6).
++
++For example, the CredentialProviderToken can be used in conjunction with a Java
++KeyStore to alleviate passwords stored in cleartext. When stored in HDFS, a single
++KeyStore can be used across an entire instance. Be aware that KeyStores stored on
++the local filesystem must be made available to all nodes in the Accumulo cluster.
++
 +=== Writing Data
 +
 +Data are written to Accumulo by creating Mutation objects that represent all the
 +changes to the columns of a single row. The changes are made atomically in the
 +TabletServer. Clients then add Mutations to a BatchWriter which submits them to
 +the appropriate TabletServers.
 +
 +Mutations can be created thus:
 +
 +[source,java]
 +----
 +Text rowID = new Text("row1");
 +Text colFam = new Text("myColFam");
 +Text colQual = new Text("myColQual");
 +ColumnVisibility colVis = new ColumnVisibility("public");
 +long timestamp = System.currentTimeMillis();
 +
 +Value value = new Value("myValue".getBytes());
 +
 +Mutation mutation = new Mutation(rowID);
 +mutation.put(colFam, colQual, colVis, timestamp, value);
 +----
 +
 +==== BatchWriter
 +The BatchWriter is highly optimized to send Mutations to multiple TabletServers
 +and automatically batches Mutations destined for the same TabletServer to
 +amortize network overhead. Care must be taken to avoid changing the contents of
 +any Object passed to the BatchWriter since it keeps objects in memory while
 +batching.
 +
 +Mutations are added to a BatchWriter thus:
 +
 +[source,java]
 +----
 +// BatchWriterConfig has reasonable defaults
 +BatchWriterConfig config = new BatchWriterConfig();
 +config.setMaxMemory(10000000L); // bytes available to batchwriter for buffering mutations
 +
 +BatchWriter writer = conn.createBatchWriter("table", config)
 +
 +writer.add(mutation);
 +
 +writer.close();
 +----
 +
 +An example of using the batch writer can be found at
 ++accumulo/docs/examples/README.batch+.
 +
 +==== ConditionalWriter
 +The ConditionalWriter enables efficient, atomic read-modify-write operations on
 +rows.  The ConditionalWriter writes special Mutations which have a list of per
 +column conditions that must all be met before the mutation is applied.  The
 +conditions are checked in the tablet server while a row lock is
 +held (Mutations written by the BatchWriter will not obtain a row
 +lock).  The conditions that can be checked for a column are equality and
 +absence.  For example a conditional mutation can require that column A is
 +absent inorder to be applied.  Iterators can be applied when checking
 +conditions.  Using iterators, many other operations besides equality and
 +absence can be checked.  For example, using an iterator that converts values
 +less than 5 to 0 and everything else to 1, its possible to only apply a
 +mutation when a column is less than 5.
 +
 +In the case when a tablet server dies after a client sent a conditional
 +mutation, its not known if the mutation was applied or not.  When this happens
 +the ConditionalWriter reports a status of UNKNOWN for the ConditionalMutation.
 +In many cases this situation can be dealt with by simply reading the row again
 +and possibly sending another conditional mutation.  If this is not sufficient,
 +then a higher level of abstraction can be built by storing transactional
 +information within a row.
 +
 +An example of using the batch writer can be found at
 ++accumulo/docs/examples/README.reservations+.
 +
 +=== Reading Data
 +
 +Accumulo is optimized to quickly retrieve the value associated with a given key, and
 +to efficiently return ranges of consecutive keys and their associated values.
 +
 +==== Scanner
 +
 +To retrieve data, Clients use a Scanner, which acts like an Iterator over
 +keys and values. Scanners can be configured to start and stop at particular keys, and
 +to return a subset of the columns available.
 +
 +[source,java]
 +----
 +// specify which visibilities we are allowed to see
 +Authorizations auths = new Authorizations("public");
 +
 +Scanner scan =
 +    conn.createScanner("table", auths);
 +
 +scan.setRange(new Range("harry","john"));
 +scan.fetchColumnFamily(new Text("attributes"));
 +
 +for(Entry<Key,Value> entry : scan) {
 +    Text row = entry.getKey().getRow();
 +    Value value = entry.getValue();
 +}
 +----
 +
 +==== Isolated Scanner
 +
 +Accumulo supports the ability to present an isolated view of rows when
 +scanning. There are three possible ways that a row could change in Accumulo :
 +
 +* a mutation applied to a table
 +* iterators executed as part of a minor or major compaction
 +* bulk import of new files
 +
 +Isolation guarantees that either all or none of the changes made by these
 +operations on a row are seen. Use the IsolatedScanner to obtain an isolated
 +view of an Accumulo table. When using the regular scanner it is possible to see
 +a non isolated view of a row. For example if a mutation modifies three
 +columns, it is possible that you will only see two of those modifications.
 +With the isolated scanner either all three of the changes are seen or none.
 +
 +The IsolatedScanner buffers rows on the client side so a large row will not
 +crash a tablet server. By default rows are buffered in memory, but the user
 +can easily supply their own buffer if they wish to buffer to disk when rows are
 +large.
 +
 +For an example, look at the following
 +
 +  examples/simple/src/main/java/org/apache/accumulo/examples/simple/isolation/InterferenceTest.java
 +
 +==== BatchScanner
 +
 +For some types of access, it is more efficient to retrieve several ranges
 +simultaneously. This arises when accessing a set of rows that are not consecutive
 +whose IDs have been retrieved from a secondary index, for example.
 +
 +The BatchScanner is configured similarly to the Scanner; it can be configured to
 +retrieve a subset of the columns available, but rather than passing a single Range,
 +BatchScanners accept a set of Ranges. It is important to note that the keys returned
 +by a BatchScanner are not in sorted order since the keys streamed are from multiple
 +TabletServers in parallel.
 +
 +[source,java]
 +----
 +ArrayList<Range> ranges = new ArrayList<Range>();
 +// populate list of ranges ...
 +
 +BatchScanner bscan =
 +    conn.createBatchScanner("table", auths, 10);
 +bscan.setRanges(ranges);
 +bscan.fetchColumnFamily("attributes");
 +
 +for(Entry<Key,Value> entry : scan) {
 +    System.out.println(entry.getValue());
 +}
 +----
 +
 +An example of the BatchScanner can be found at
 ++accumulo/docs/examples/README.batch+.
 +
 +=== Proxy
 +
 +The proxy API allows the interaction with Accumulo with languages other than Java.
 +A proxy server is provided in the codebase and a client can further be generated.
 +
 +==== Prequisites
 +
 +The proxy server can live on any node in which the basic client API would work. That
 +means it must be able to communicate with the Master, ZooKeepers, NameNode, and the
 +DataNodes. A proxy client only needs the ability to communicate with the proxy server.
 +
 +
 +==== Configuration
 +
 +The configuration options for the proxy server live inside of a properties file. At
 +the very least, you need to supply the following properties:
 +
 +  protocolFactory=org.apache.thrift.protocol.TCompactProtocol$Factory
 +  tokenClass=org.apache.accumulo.core.client.security.tokens.PasswordToken
 +  port=42424
 +  instance=test
 +  zookeepers=localhost:2181
 +
 +You can find a sample configuration file in your distribution:
 +
 +  $ACCUMULO_HOME/proxy/proxy.properties.
 +
 +This sample configuration file further demonstrates an ability to back the proxy server
 +by MockAccumulo or the MiniAccumuloCluster.
 +
 +==== Running the Proxy Server
 +
 +After the properties file holding the configuration is created, the proxy server
 +can be started using the following command in the Accumulo distribution (assuming
 +your properties file is named +config.properties+):
 +
 +  $ACCUMULO_HOME/bin/accumulo proxy -p config.properties
 +
 +==== Creating a Proxy Client
 +
 +Aside from installing the Thrift compiler, you will also need the language-specific library
 +for Thrift installed to generate client code in that language. Typically, your operating
 +system's package manager will be able to automatically install these for you in an expected
 +location such as +/usr/lib/python/site-packages/thrift+.
 +
 +You can find the thrift file for generating the client:
 +
 +  $ACCUMULO_HOME/proxy/proxy.thrift.
 +
 +After a client is generated, the port specified in the configuration properties above will
be
 +used to connect to the server.
 +
 +==== Using a Proxy Client
 +
 +The following examples have been written in Java and the method signatures may be
 +slightly different depending on the language specified when generating client with
 +the Thrift compiler. After initiating a connection to the Proxy (see Apache Thrift's
 +documentation for examples of connecting to a Thrift service), the methods on the
 +proxy client will be available. The first thing to do is log in:
 +
 +[source,java]
 +Map password = new HashMap<String,String>();
 +password.put("password", "secret");
 +ByteBuffer token = client.login("root", password);
 +
 +Once logged in, the token returned will be used for most subsequent calls to the client.
 +Let's create a table, add some data, scan the table, and delete it.
 +
 +
 +First, create a table.
 +
 +[source,java]
 +client.createTable(token, "myTable", true, TimeType.MILLIS);
 +
 +
 +Next, add some data:
 +
 +[source,java]
 +----
 +// first, create a writer on the server
 +String writer = client.createWriter(token, "myTable", new WriterOptions());
 +
 +// build column updates
 +Map<ByteBuffer, List<ColumnUpdate> cells> cellsToUpdate = //...
 +
 +// send updates to the server
 +client.updateAndFlush(writer, "myTable", cellsToUpdate);
 +
 +client.closeWriter(writer);
 +----
 +
 +
 +Scan for the data and batch the return of the results on the server:
 +
 +[source,java]
 +----
 +String scanner = client.createScanner(token, "myTable", new ScanOptions());
 +ScanResult results = client.nextK(scanner, 100);
 +
 +for(KeyValue keyValue : results.getResultsIterator()) {
 +  // do something with results
 +}
 +
 +client.closeScanner(scanner);
 +----


Mime
View raw message