accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mwa...@apache.org
Subject [14/16] accumulo-website git commit: ACCUMULO-4630 Refactored documentation for website
Date Mon, 22 May 2017 17:30:03 GMT
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/7cc70b2e/_docs-unreleased/administration/overview.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/administration/overview.md b/_docs-unreleased/administration/overview.md
new file mode 100644
index 0000000..39ff163
--- /dev/null
+++ b/_docs-unreleased/administration/overview.md
@@ -0,0 +1,1151 @@
+---
+title: Overview
+category: administration
+order: 1
+---
+
+## Hardware
+
+Because we are running essentially two or three systems simultaneously layered
+across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
+consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
+at least one core and 2 - 4 GB each.
+
+One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
+typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.
+
+It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
+each, but in this case it is recommended to only run up to two processes per
+machine -- i.e. DataNode and TabletServer or DataNode and MapReduce worker but
+not all three. The constraint here is having enough available heap space for all the
+processes on a machine.
+
+## Network
+
+Accumulo communicates via remote procedure calls over TCP/IP for both passing
+data and control messages. In addition, Accumulo uses HDFS clients to
+communicate with HDFS. To achieve good ingest and query performance, sufficient
+network bandwidth must be available between any two machines.
+
+In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
+use the following default ports. Please make sure that they are open, or change
+their value in accumulo-site.xml.
+
+|Port | Description | Property Name
+|-----|-------------|--------------
+|4445 | Shutdown Port (Accumulo MiniCluster) | n/a
+|4560 | Accumulo monitor (for centralized log display) | monitor.port.log4j
+|9995 | Accumulo HTTP monitor | monitor.port.client
+|9997 | Tablet Server | tserver.port.client
+|9998 | Accumulo GC | gc.port.client
+|9999 | Master Server | master.port.client
+|12234 | Accumulo Tracer | trace.port.client
+|42424 | Accumulo Proxy Server | n/a
+|10001 | Master Replication service | master.replication.coordinator.port
+|10002 | TabletServer Replication service | replication.receipt.service.port
+
+In addition, the user can provide `0` and an ephemeral port will be chosen instead. This
+ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
+use `0` instead of an explicit value, should, in most cases, work around any issues of
+running multiple distinct Accumulo instances (or any other process which tries to use the
+same default ports) on the same hardware. Finally, the *.port.client properties will work
+with the port range syntax (M-N) allowing the user to specify a range of ports for the
+service to attempt to bind. The ports in the range will be tried in a 1-up manner starting
+at the low end of the range to, and including, the high end of the range.
+
+## Installation
+
+Download a binary distribution of Accumulo and install it to a directory on a disk with
+sufficient space:
+
+    cd <install directory>
+    tar xzf accumulo-X.Y.Z-bin.tar.gz   # Replace 'X.Y.Z' with your Accumulo version
+    cd accumulo-X.Y.Z
+
+Repeat this step on each machine in your cluster. Typically, the same `<install directory>`
+is chosen for all machines in the cluster.
+
+There are four scripts in the `bin/` directory that are used to manage Accumulo:
+
+1. `accumulo` - Runs Accumulo command-line tools and starts Accumulo processes
+2. `accumulo-service` - Runs Accumulo processes as services
+3. `accumulo-cluster` - Manages Accumulo cluster on a single node or several nodes
+4. `accumulo-util` - Accumulo utilities for creating configuration, native libraries, etc.
+
+These scripts will be used in the remaining instructions to configure and run Accumulo.
+
+## Dependencies
+
+Accumulo requires HDFS and ZooKeeper to be configured and running
+before starting. Password-less SSH should be configured between at least the
+Accumulo master and TabletServer machines. It is also a good idea to run Network
+Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of
+sync, which can cause problems with automatically timestamped data.
+
+## Configuration
+
+The Accumulo tarball contains a `conf/` directory where Accumulo looks for configuration. If you
+installed Accumulo using downstream packaging, the `conf/` could be something else like
+`/etc/accumulo/`.
+
+Before starting Accumulo, the configuration files `accumulo-env.sh` and `accumulo-site.xml` must
+exist in `conf/` and be properly configured. If you are using `accumulo-cluster` to launch
+a cluster, the `conf/` directory must also contain hosts file for Accumulo services (i.e `gc`,
+`masters`, `monitor`, `tservers`, `tracers`). You can either create these files manually or run
+`accumulo-cluster create-config`.
+
+Logging is configured in `accumulo-env.sh` to use three log4j configuration files in `conf/`. The
+file used depends on the Accumulo command or service being run. Logging for most Accumulo services
+(i.e Master, TabletServer, Garbage Collector) is configured by `log4j-service.properties` except for
+the Monitor which is configured by `log4j-monitor.properties`. All Accumulo commands (i.e `init`,
+`shell`, etc) are configured by `log4j.properties`.
+
+### Configure accumulo-env.sh
+
+Accumulo needs to know where to find the software it depends on. Edit accumulo-env.sh
+and specify the following:
+
+1. Enter the location of Hadoop for `$HADOOP_PREFIX`
+2. Enter the location of ZooKeeper for `$ZOOKEEPER_HOME`
+3. Optionally, choose a different location for Accumulo logs using `$ACCUMULO_LOG_DIR`
+
+Accumulo uses `HADOOP_PREFIX` and `ZOOKEEPER_HOME` to locate Hadoop and Zookeeper jars
+and add them the `CLASSPATH` variable. If you are running a vendor-specific release of Hadoop
+or Zookeeper, you may need to change how your `CLASSPATH` is built in `accumulo-env.sh`. If
+Accumulo has problems later on finding jars, run `accumulo classpath -d` to debug and print
+Accumulo's classpath.
+
+You may want to change the default memory settings for Accumulo's TabletServer which are
+by set in the `JAVA_OPTS` settings for 'tservers' in `accumulo-env.sh`. Note the
+syntax is that of the Java JVM command line options. This value should be less than the
+physical memory of the machines running TabletServers.
+
+There are similar options for the master's memory usage and the garbage collector
+process. Reduce these if they exceed the physical RAM of your hardware and
+increase them, within the bounds of the physical RAM, if a process fails because of
+insufficient memory.
+
+Note that you will be specifying the Java heap space in accumulo-env.sh. You should
+make sure that the total heap space used for the Accumulo tserver and the Hadoop
+DataNode and TaskTracker is less than the available memory on each worker node in
+the cluster. On large clusters, it is recommended that the Accumulo master, Hadoop
+NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate
+machines to allow them to use more heap space. If you are running these on the
+same machine on a small cluster, likewise make sure their heap space settings fit
+within the available memory.
+
+### Native Map
+
+The tablet server uses a data structure called a MemTable to store sorted key/value
+pairs in memory when they are first received from the client. When a minor compaction
+occurs, this data structure is written to HDFS. The MemTable will default to using
+memory in the JVM but a JNI version, called the native map, can be used to significantly
+speed up performance by utilizing the memory space of the native operating system. The
+native map also avoids the performance implications brought on by garbage collection
+in the JVM by causing it to pause much less frequently.
+
+#### Building
+
+32-bit and 64-bit Linux and Mac OS X versions of the native map can be built by executing
+`accumulo-util build-native`. If your system's default compiler options are insufficient,
+you can add additional compiler options to the command line, such as options for the
+architecture. These will be passed to the Makefile in the environment variable `USERFLAGS`.
+
+Examples:
+
+    accumulo-util build-native
+    accumulo-util build-native -m32
+
+After building the native map from the source, you will find the artifact in
+`lib/native`. Upon starting up, the tablet server will look
+in this directory for the map library. If the file is renamed or moved from its
+target directory, the tablet server may not be able to find it. The system can
+also locate the native maps shared library by setting `LD_LIBRARY_PATH`
+(or `DYLD_LIBRARY_PATH` on Mac OS X) in `accumulo-env.sh`.
+
+#### Native Maps Configuration
+
+As mentioned, Accumulo will use the native libraries if they are found in the expected
+location and `tserver.memory.maps.native.enabled` is set to `true` (which is the default).
+Using the native maps over JVM Maps nets a noticeable improvement in ingest rates; however,
+certain configuration variables are important to modify when increasing the size of the
+native map.
+
+To adjust the size of the native map, increase the value of `tserver.memory.maps.max`.
+By default, the maximum size of the native map is 1GB. When increasing this value, it is
+also important to adjust the values of `table.compaction.minor.logs.threshold` and
+`tserver.walog.max.size`. `table.compaction.minor.logs.threshold` is the maximum
+number of write-ahead log files that a tablet can reference before they will be automatically
+minor compacted. `tserver.walog.max.size` is the maximum size of a write-ahead log.
+
+The maximum size of the native maps for a server should be less than the product
+of the write-ahead log maximum size and minor compaction threshold for log files:
+
+`$table.compaction.minor.logs.threshold * $tserver.walog.max.size >= $tserver.memory.maps.max`
+
+This formula ensures that minor compactions won't be automatically triggered before the native
+maps can be completely saturated.
+
+Subsequently, when increasing the size of the write-ahead logs, it can also be important
+to increase the HDFS block size that Accumulo uses when creating the files for the write-ahead log.
+This is controlled via `tserver.wal.blocksize`. A basic recommendation is that when
+`tserver.walog.max.size` is larger than 2GB in size, set `tserver.wal.blocksize` to 2GB.
+Increasing the block size to a value larger than 2GB can result in decreased write
+performance to the write-ahead log file which will slow ingest.
+
+### Cluster Specification
+
+If you are using `accumulo-cluster` to start a cluster, configure the following on the
+machine that will serve as the Accumulo master:
+
+1. Write the IP address or domain name of the Accumulo Master to the `conf/masters` file.
+2. Write the IP addresses or domain name of the machines that will be TabletServers in `conf/tservers`, one per line.
+
+Note that if using domain names rather than IP addresses, DNS must be configured
+properly for all machines participating in the cluster. DNS can be a confusing source
+of errors.
+
+### Configure accumulo-site.xml
+
+Specify appropriate values for the following settings in `accumulo-site.xml`:
+
+```xml
+<property>
+    <name>instance.zookeeper.host</name>
+    <value>zooserver-one:2181,zooserver-two:2181</value>
+    <description>list of zookeeper servers</description>
+</property>
+```
+
+This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate
+settings between processes and helps finalize TabletServer failure.
+
+```xml
+<property>
+    <name>instance.secret</name>
+    <value>DEFAULT</value>
+</property>
+```
+
+The instance needs a secret to enable secure communication between servers. Configure your
+secret and make sure that the `accumulo-site.xml` file is not readable to other users.
+For alternatives to storing the `instance.secret` in plaintext, please read the
+`Sensitive Configuration Values` section.
+
+Some settings can be modified via the Accumulo shell and take effect immediately, but
+some settings require a process restart to take effect. See the [configuration management][config-mgmt]
+documentation for details.
+
+### Hostnames in configuration files
+
+Accumulo has a number of configuration files which can contain references to other hosts in your
+network. All of the "host" configuration files for Accumulo (`gc`, `masters`, `tservers`, `monitor`,
+`tracers`) as well as `instance.volumes` in accumulo-site.xml must contain some host reference.
+
+While IP address, short hostnames, or fully qualified domain names (FQDN) are all technically valid, it
+is good practice to always use FQDNs for both Accumulo and other processes in your Hadoop cluster.
+Failing to consistently use FQDNs can have unexpected consequences in how Accumulo uses the FileSystem.
+
+A common way for this problem can be observed is via applications that use Bulk Ingest. The Accumulo
+Master coordinates moving the input files to Bulk Ingest to an Accumulo-managed directory. However,
+Accumulo cannot safely move files across different Hadoop FileSystems. This is problematic because
+Accumulo also cannot make reliable assertions across what is the same FileSystem which is specified
+with different names. Naively, while 127.0.0.1:8020 might be a valid identifier for an HDFS instance,
+Accumulo identifies `localhost:8020` as a different HDFS instance than `127.0.0.1:8020`.
+
+### Deploy Configuration
+
+Copy accumulo-env.sh and accumulo-site.xml from the `conf/` directory on the master to all Accumulo
+tablet servers.  The "host" configuration files files `accumulo-cluster` only need to be on servers
+where that command is run.
+
+### Sensitive Configuration Values
+
+Accumulo has a number of properties that can be specified via the accumulo-site.xml
+file which are sensitive in nature, instance.secret and trace.token.property.password
+are two common examples. Both of these properties, if compromised, have the ability
+to result in data being leaked to users who should not have access to that data.
+
+In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves as a common
+implementation to abstract away the storage and retrieval of passwords from plaintext
+storage in configuration files. Any Property marked with the `Sensitive` annotation
+is a candidate for use with these CredentialProviders. For version of Hadoop which lack
+these classes, the feature will just be unavailable for use.
+
+A comma separated list of CredentialProviders can be configured using the Accumulo Property
+`general.security.credential.provider.paths`. Each configured URL will be consulted
+when the Configuration object for accumulo-site.xml is accessed.
+
+### Using a JavaKeyStoreCredentialProvider for storage
+
+One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore CredentialProvider.
+Each entry in the KeyStore is the Accumulo Property key name. For example, to store the
+`instance.secret`, the following command can be used:
+
+    hadoop credential create instance.secret --provider jceks://file/etc/accumulo/conf/accumulo.jceks
+
+The command will then prompt you to enter the secret to use and create a keystore in: 
+
+    /path/to/accumulo/conf/accumulo.jceks
+
+Then, accumulo-site.xml must be configured to use this KeyStore as a CredentialProvider:
+
+```xml
+<property>
+    <name>general.security.credential.provider.paths</name>
+    <value>jceks://file/path/to/accumulo/conf/accumulo.jceks</value>
+</property>
+```
+
+This configuration will then transparently extract the `instance.secret` from
+the configured KeyStore and alleviates a human readable storage of the sensitive
+property.
+
+A KeyStore can also be stored in HDFS, which will make the KeyStore readily available to
+all Accumulo servers. If the local filesystem is used, be aware that each Accumulo server
+will expect the KeyStore in the same location.
+
+### Client Configuration
+
+In version 1.6.0, Accumulo included a new type of configuration file known as a client
+configuration file. One problem with the traditional "site.xml" file that is prevalent
+through Hadoop is that it is a single file used by both clients and servers. This makes
+it very difficult to protect secrets that are only meant for the server processes while
+allowing the clients to connect to the servers.
+
+The client configuration file is a subset of the information stored in accumulo-site.xml
+meant only for consumption by clients of Accumulo. By default, Accumulo checks a number
+of locations for a client configuration by default:
+
+* `/path/to/accumulo/conf/client.conf`
+* `/etc/accumulo/client.conf`
+* `/etc/accumulo/conf/client.conf`
+* `~/.accumulo/config`
+
+These files are [Java Properties files](https://en.wikipedia.org/wiki/.properties). These files
+can currently contain information about ZooKeeper servers, RPC properties (such as SSL or SASL
+connectors), distributed tracing properties. Valid properties are defined by the [ClientProperty](https://github.com/apache/accumulo/blob/f1d0ec93d9f13ff84844b5ac81e4a7b383ced467/core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java#L54)
+enum contained in the client API.
+
+#### Custom Table Tags
+
+Accumulo has the ability for users to add custom tags to tables.  This allows
+applications to set application-level metadata about a table.  These tags can be
+anything from a table description, administrator notes, date created, etc.
+This is done by naming and setting a property with a prefix `table.custom.*`.
+
+Currently, table properties are stored in ZooKeeper. This means that the number
+and size of custom properties should be restricted on the order of 10's of properties
+at most without any properties exceeding 1MB in size. ZooKeeper's performance can be
+very sensitive to an excessive number of nodes and the sizes of the nodes. Applications
+which leverage the user of custom properties should take these warnings into
+consideration. There is no enforcement of these warnings via the API.
+
+#### Configuring the ClassLoader
+
+Accumulo builds its Java classpath in `accumulo-env.sh`.  After an Accumulo application has started, it will load classes from the locations
+specified in the deprecated `general.classpaths` property. Additionally, Accumulo will load classes from the locations specified in the
+`general.dynamic.classpaths` property and will monitor and reload them if they change. The reloading  feature is useful during the development
+and testing of iterators as new or modified iterator classes can be deployed to Accumulo without having to restart the database.
+/
+Accumulo also has an alternate configuration for the classloader which will allow it to load classes from remote locations. This mechanism
+uses Apache Commons VFS which enables locations such as http and hdfs to be used. This alternate configuration also uses the
+`general.classpaths` property in the same manner described above. It differs in that you need to configure the
+`general.vfs.classpaths` property instead of the `general.dynamic.classpath` property. As in the default configuration, this alternate
+configuration will also monitor the vfs locations for changes and reload if necessary.
+
+The Accumulo classpath can be viewed in human readable format by running `accumulo classpath -d`.
+
+##### ClassLoader Contexts
+
+With the addition of the VFS based classloader, we introduced the notion of classloader contexts. A context is identified
+by a name and references a set of locations from which to load classes and can be specified in the accumulo-site.xml file or added
+using the `config` command in the shell. Below is an example for specify the app1 context in the accumulo-site.xml file:
+
+```xml
+<property>
+  <name>general.vfs.context.classpath.app1</name>
+  <value>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar</value>
+  <description>Application A classpath, loads jars from HDFS and local file system</description>
+</property>
+```
+
+The default behavior follows the Java ClassLoader contract in that classes, if they exists, are loaded from the parent classloader first.
+You can override this behavior by delegating to the parent classloader after looking in this classloader first. An example of this
+configuration is:
+
+```xml
+<property>
+  <name>general.vfs.context.classpath.app1.delegation=post</name>
+  <value>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar</value>
+  <description>Application A classpath, loads jars from HDFS and local file system</description>
+</property>
+```
+
+To use contexts in your application you can set the `table.classpath.context` on your tables or use the `setClassLoaderContext()` method on Scanner
+and BatchScanner passing in the name of the context, app1 in the example above. Setting the property on the table allows your minc, majc, and scan 
+iterators to load classes from the locations defined by the context. Passing the context name to the scanners allows you to override the table setting
+to load only scan time iterators from a different location. 
+
+## Initialization
+
+Accumulo must be initialized to create the structures it uses internally to locate
+data across the cluster. HDFS is required to be configured and running before
+Accumulo can be initialized.
+
+Once HDFS is started, initialization can be performed by executing
+`accumulo init` . This script will prompt for a name
+for this instance of Accumulo. The instance name is used to identify a set of tables
+and instance-specific settings. The script will then write some information into
+HDFS so Accumulo can start properly.
+
+The initialization script will prompt you to set a root password. Once Accumulo is
+initialized it can be started.
+
+## Running
+
+### Starting Accumulo
+
+Make sure Hadoop is configured on all of the machines in the cluster, including
+access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running.
+Make sure ZooKeeper is configured and running on at least one machine in the
+cluster.
+Start Accumulo using `accumulo-cluster start`.
+
+To verify that Accumulo is running, check the [Accumulo monitor][monitor].
+In addition, the Shell can provide some information about the status of tables via reading the metadata tables.
+
+### Stopping Accumulo
+
+To shutdown cleanly, run `accumulo-cluster stop` and the master will orchestrate the
+shutdown of all the tablet servers. Shutdown waits for all minor compactions to finish, so it may
+take some time for particular configurations.
+
+### Adding a Tablet Server
+
+Update your `conf/tservers` file to account for the addition.
+
+Next, ssh to each of the hosts you want to add and run:
+
+    accumulo-service tserver start
+
+Make sure the host in question has the new configuration, or else the tablet
+server won't start; at a minimum this needs to be on the host(s) being added,
+but in practice it's good to ensure consistent configuration across all nodes.
+
+### Decomissioning a Tablet Server
+
+If you need to take a node out of operation, you can trigger a graceful shutdown of a tablet
+server. Accumulo will automatically rebalance the tablets across the available tablet servers.
+
+    accumulo admin stop <host(s)> {<host> ...}
+
+Alternatively, you can ssh to each of the hosts you want to remove and run:
+
+    accumulo-service tserver stop
+
+Be sure to update your `conf/tservers` file to
+account for the removal of these hosts. Bear in mind that the monitor will not re-read the
+tservers file automatically, so it will report the decommissioned servers as down; it's
+recommended that you restart the monitor so that the node list is up to date.
+
+The steps described to decommission a node can also be used (without removal of the host
+from the `conf/tservers` file) to gracefully stop a node. This will
+ensure that the tabletserver is cleanly stopped and recovery will not need to be performed
+when the tablets are re-hosted.
+
+### Restarting process on a node
+
+Occasionally, it might be necessary to restart the processes on a specific node. In addition
+to the `accumulo-cluster` script, Accumulo has a `accumulo-service` script that
+can be use to start/stop processes on a node.
+
+#### A note on rolling restarts
+
+For sufficiently large Accumulo clusters, restarting multiple TabletServers within a short window can place significant 
+load on the Master server.  If slightly lower availability is acceptable, this load can be reduced by globally setting 
+`table.suspend.duration` to a positive value.  
+
+With `table.suspend.duration` set to, say, `5m`, Accumulo will wait 
+for 5 minutes for any dead TabletServer to return before reassigning that TabletServer's responsibilities to other TabletServers.
+If the TabletServer returns to the cluster before the specified timeout has elapsed, Accumulo will assign the TabletServer 
+its original responsibilities.
+
+It is important not to choose too large a value for `table.suspend.duration`, as during this time, all scans against the 
+data that TabletServer had hosted will block (or time out).
+
+### Running multiple TabletServers on a single node
+
+With very powerful nodes, it may be beneficial to run more than one TabletServer on a given
+node. This decision should be made carefully and with much deliberation as Accumulo is designed
+to be able to scale to using 10's of GB of RAM and 10's of CPU cores.
+
+Accumulo TabletServers bind certain ports on the host to accommodate remote procedure calls to/from
+other nodes. Running more than one TabletServer on a host requires that you set the environment variable
+`ACCUMULO_SERVICE_INSTANCE` to an instance number (i.e 1, 2) for each instance that is started. Also, set
+these properties in `accumulo-site.xml`:
+
+```xml
+  <property>
+    <name>tserver.port.search</name>
+    <value>true</value>
+  </property>
+  <property>
+    <name>replication.receipt.service.port</name>
+    <value>0</value>
+  </property>
+```
+
+## Monitoring
+
+### Accumulo Monitor
+
+The Accumulo Monitor provides an interface for monitoring the status and health of
+Accumulo components. The Accumulo Monitor provides a web UI for accessing this information at
+`http://_monitorhost_:9995/`.
+
+Things highlighted in yellow may be in need of attention.
+If anything is highlighted in red on the monitor page, it is something that definitely needs attention.
+
+The Overview page contains some summary information about the Accumulo instance, including the version, instance name, and instance ID.
+There is a table labeled Accumulo Master with current status, a table listing the active Zookeeper servers, and graphs displaying various metrics over time.
+These include ingest and scan performance and other useful measurements.
+
+The Master Server, Tablet Servers, and Tables pages display metrics grouped in different ways (e.g. by tablet server or by table).
+Metrics typically include number of entries (key/value pairs), ingest and query rates.
+The number of running scans, major and minor compactions are in the form _number_running_ (_number_queued_).
+Another important metric is hold time, which is the amount of time a tablet has been waiting but unable to flush its memory in a minor compaction.
+
+The Server Activity page graphically displays tablet server status, with each server represented as a circle or square.
+Different metrics may be assigned to the nodes' color and speed of oscillation.
+The Overall Avg metric is only used on the Server Activity page, and represents the average of all the other metrics (after normalization).
+Similarly, the Overall Max metric picks the metric with the maximum normalized value.
+
+The Garbage Collector page displays a list of garbage collection cycles, the number of files found of each type (including deletion candidates in use and files actually deleted), and the length of the deletion cycle.
+The Traces page displays data for recent traces performed (see the following section for information on [tracing][tracing]).
+The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo processes.
+Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.
+
+### SSL
+
+SSL may be enabled for the monitor page by setting the following properties in the `accumulo-site.xml` file:
+
+    monitor.ssl.keyStore
+    monitor.ssl.keyStorePassword
+    monitor.ssl.trustStore
+    monitor.ssl.trustStorePassword
+
+If the Accumulo conf directory has been configured (in particular the `accumulo-env.sh` file must be set up), the 
+`accumulo-util gen-monitor-cert` command can be used to create the keystore and truststore files with random passwords. The command
+will print out the properties that need to be added to the `accumulo-site.xml` file. The stores can also be generated manually with the
+Java `keytool` command, whose usage can be seen in the `accumulo-util` script.
+
+If desired, the SSL ciphers allowed for connections can be controlled via the following properties in `accumulo-site.xml`:
+
+    monitor.ssl.include.ciphers
+    monitor.ssl.exclude.ciphers
+
+If SSL is enabled, the monitor URL can only be accessed via https.
+This also allows you to access the Accumulo shell through the monitor page.
+The left navigation bar will have a new link to Shell.
+An Accumulo user name and password must be entered for access to the shell.
+
+## Metrics
+
+Accumulo can expose metrics through a legacy metrics library and using the Hadoop Metrics2 library.
+
+### Legacy Metrics
+
+Accumulo has a legacy metrics library that can be exposes metrics using JMX endpoints or file-based logging. These metrics can
+be enabled by setting `general.legacy.metrics` to `true` in `accumulo-site.xml` and placing the `accumulo-metrics.xml`
+configuration file on the classpath (which is typically done by placing the file in the `conf/` directory). A template for
+`accumulo-metrics.xml` can be found in `conf/templates` of the Accumulo tarball.
+
+### Hadoop Metrics2
+
+Hadoop Metrics2 is a library which allows for routing of metrics generated by registered MetricsSources to
+configured MetricsSinks. Examples of sinks that are implemented by Hadoop include file-based logging, Graphite and Ganglia.
+All metric sources are exposed via JMX when using Metrics2.
+
+Metrics2 is configured by examining the classpath for a file that matches `hadoop-metrics2*.properties`. The Accumulo tarball 
+contains an example `hadoop-metrics2-accumulo.properties` file in `conf/templates` which can be copied to `conf/` to place
+on classpath. This file is used to enable file, Graphite or Ganglia sinks (some minimal configuration required for Graphite
+and Ganglia). Because the Hadoop configuration is also on the Accumulo classpath, be sure that you do not have multiple
+Metrics2 configuration files. It is recommended to consolidate metrics in a single properties file in a central location to
+remove ambiguity. The contents of `hadoop-metrics2-accumulo.properties` can be added to a central `hadoop-metrics2.properties`
+in `$HADOOP_CONF_DIR`.
+
+As a note for configuring the file sink, the provided path should be absolute. A relative path or file name will be created relative
+to the directory in which the Accumulo process was started. External tools, such as logrotate, can be used to prevent these files
+from growing without bound.
+
+Each server process should have log messages from the Metrics2 library about the sinks that were created. Be sure to check
+the Accumulo processes log files when debugging missing metrics output.
+
+For additional information on configuring Metrics2, visit the [Javadoc page for Metrics2](https://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html).
+
+## Tracing
+
+It can be difficult to determine why some operations are taking longer
+than expected. For example, you may be looking up items with very low
+latency, but sometimes the lookups take much longer. Determining the
+cause of the delay is difficult because the system is distributed, and
+the typical lookup is fast.
+
+Accumulo has been instrumented to record the time that various
+operations take when tracing is turned on. The fact that tracing is
+enabled follows all the requests made on behalf of the user throughout
+the distributed infrastructure of accumulo, and across all threads of
+execution.
+
+These time spans will be inserted into the `trace` table in
+Accumulo. You can browse recent traces from the Accumulo monitor
+page. You can also read the `trace` table directly like any
+other table.
+
+The design of Accumulo's distributed tracing follows that of [Google's Dapper](http://research.google.com/pubs/pub36356.html).
+
+### Tracers
+
+To collect traces, Accumulo needs at least one tracer server running. If you are using `accumulo-cluster` to start your cluster,
+configure your server in `conf/tracers`. The server collects traces from clients and writes them to the `trace` table. The Accumulo
+user that the tracer connects to Accumulo with can be configured with the following properties (see the [configuration management][config-mgmt] 
+page for setting Accumulo server properties)
+
+    trace.user
+    trace.token.property.password
+
+Other tracer configuration properties include
+
+    trace.port.client - port tracer listens on
+    trace.table - table tracer writes to
+    trace.zookeeper.path - zookeeper path where tracers register
+
+The zookeeper path is configured to /tracers by default.  If
+multiple Accumulo instances are sharing the same ZooKeeper
+quorum, take care to configure Accumulo with unique values for
+this property.
+
+### Configuring Tracing
+
+Traces are collected via SpanReceivers. The default SpanReceiver
+configured is org.apache.accumulo.core.trace.ZooTraceClient, which
+sends spans to an Accumulo Tracer process, as discussed in the
+previous section. This default can be changed to a different span
+receiver, or additional span receivers can be added in a
+comma-separated list, by modifying the property
+
+    trace.span.receivers
+
+Individual span receivers may require their own configuration
+parameters, which are grouped under the trace.span.receiver.*
+prefix.  ZooTraceClient uses the following properties.  The first
+three properties are populated from other Accumulo properties,
+while the remaining ones should be prefixed with
+trace.span.receiver. when set in the Accumulo configuration.
+
+    tracer.zookeeper.host - populated from instance.zookeepers
+    tracer.zookeeper.timeout - populated from instance.zookeeper.timeout
+    tracer.zookeeper.path - populated from trace.zookeeper.path
+    tracer.send.timer.millis - timer for flushing send queue (in ms, default 1000)
+    tracer.queue.size - max queue size (default 5000)
+    tracer.span.min.ms - minimum span length to store (in ms, default 1)
+
+Note that to configure an Accumulo client for tracing, including
+the Accumulo shell, the client configuration must be given the same
+trace.span.receivers, trace.span.receiver.*, and trace.zookeeper.path
+properties as the servers have.
+
+Hadoop can also be configured to send traces to Accumulo, as of
+Hadoop 2.6.0, by setting properties in Hadoop's core-site.xml
+file.  Instead of using the trace.span.receiver.* prefix, Hadoop
+uses hadoop.htrace.*.  The Hadoop configuration does not have
+access to Accumulo's properties, so the
+hadoop.htrace.tracer.zookeeper.host property must be specified.
+The zookeeper timeout defaults to 30000 (30 seconds), and the
+zookeeper path defaults to /tracers.  An example of configuring
+Hadoop to send traces to ZooTraceClient is
+
+```xml
+  <property>
+    <name>hadoop.htrace.spanreceiver.classes</name>
+    <value>org.apache.accumulo.core.trace.ZooTraceClient</value>
+  </property>
+  <property>
+    <name>hadoop.htrace.tracer.zookeeper.host</name>
+    <value>zookeeperHost:2181</value>
+  </property>
+  <property>
+    <name>hadoop.htrace.tracer.zookeeper.path</name>
+    <value>/tracers</value>
+  </property>
+  <property>
+    <name>hadoop.htrace.tracer.span.min.ms</name>
+    <value>1</value>
+  </property>
+```
+
+The accumulo-core, accumulo-tracer, accumulo-fate and libthrift
+jars must also be placed on Hadoop's classpath.
+
+##### Adding additional SpanReceivers
+
+[Zipkin](https://github.com/openzipkin/zipkin) has a SpanReceiver supported by HTrace and popularized by Twitter
+that users looking for a more graphical trace display may opt to use.
+The following steps configure Accumulo to use `org.apache.htrace.impl.ZipkinSpanReceiver`
+in addition to the Accumulo's default ZooTraceClient, and they serve as a template
+for adding any SpanReceiver to Accumulo:
+
+1. Add the Jar containing the ZipkinSpanReceiver class file to the
+`lib/` directory.  It is critical that the Jar is placed in
+`lib/` and NOT in `lib/ext/` so that the new SpanReceiver class
+is visible to the same class loader of htrace-core.
+
+2. Add the following to `accumulo-site.xml`:
+
+        <property>
+          <name>trace.span.receivers</name>
+          <value>org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver</value>
+        </property>
+
+3. Restart your Accumulo tablet servers.
+
+In order to use ZipkinSpanReceiver from a client as well as the Accumulo server,
+
+1. Ensure your client can see the ZipkinSpanReceiver class at runtime. For Maven projects,
+this is easily done by adding to your client's pom.xml (taking care to specify a good version)
+
+        <dependency>
+          <groupId>org.apache.htrace</groupId>
+          <artifactId>htrace-zipkin</artifactId>
+          <version>3.1.0-incubating</version>
+          <scope>runtime</scope>
+        </dependency>
+
+2. Add the following to your client configuration.
+
+        trace.span.receivers=org.apache.accumulo.tracer.ZooTraceClient,org.apache.htrace.impl.ZipkinSpanReceiver
+
+3. Instrument your client as in the next section.
+
+Your SpanReceiver may require additional properties, and if so these should likewise
+be placed in the ClientConfiguration (if applicable) and Accumulo's `accumulo-site.xml`.
+Two such properties for ZipkinSpanReceiver, listed with their default values, are
+
+```xml
+  <property>
+    <name>trace.span.receiver.zipkin.collector-hostname</name>
+    <value>localhost</value>
+  </property>
+  <property>
+    <name>trace.span.receiver.zipkin.collector-port</name>
+    <value>9410</value>
+  </property>
+```
+
+#### Instrumenting a Client
+
+Tracing can be used to measure a client operation, such as a scan, as
+the operation traverses the distributed system. To enable tracing for
+your application call
+
+```java
+import org.apache.accumulo.core.trace.DistributedTrace;
+...
+DistributedTrace.enable(hostname, "myApplication");
+// do some tracing
+...
+DistributedTrace.disable();
+```
+
+Once tracing has been enabled, a client can wrap an operation in a trace.
+
+```java
+import org.apache.htrace.Sampler;
+import org.apache.htrace.Trace;
+import org.apache.htrace.TraceScope;
+...
+TraceScope scope = Trace.startSpan("Client Scan", Sampler.ALWAYS);
+BatchScanner scanner = conn.createBatchScanner(...);
+// Configure your scanner
+for (Entry entry : scanner) {
+}
+scope.close();
+```
+
+The user can create additional Spans within a Trace.
+
+The sampler (such as `Sampler.ALWAYS`) for the trace should only be specified with a top-level span,
+and subsequent spans will be collected depending on whether that first span was sampled.
+Don't forget to specify a Sampler at the top-level span
+because the default Sampler only samples when part of a pre-existing trace,
+which will never occur in a client that never specifies a Sampler.
+
+```java
+TraceScope scope = Trace.startSpan("Client Update", Sampler.ALWAYS);
+...
+TraceScope readScope = Trace.startSpan("Read");
+...
+readScope.close();
+...
+TraceScope writeScope = Trace.startSpan("Write");
+...
+writeScope.close();
+scope.close();
+```
+
+Like Dapper, Accumulo tracing supports user defined annotations to associate additional data with a Trace.
+Checking whether currently tracing is necessary when using a sampler other than Sampler.ALWAYS.
+
+```java
+...
+int numberOfEntriesRead = 0;
+TraceScope readScope = Trace.startSpan("Read");
+// Do the read, update the counter
+...
+if (Trace.isTracing)
+  readScope.getSpan().addKVAnnotation("Number of Entries Read".getBytes(StandardCharsets.UTF_8),
+      String.valueOf(numberOfEntriesRead).getBytes(StandardCharsets.UTF_8));
+```
+
+It is also possible to add timeline annotations to your spans.
+This associates a string with a given timestamp between the start and stop times for a span.
+
+```java
+...
+writeScope.getSpan().addTimelineAnnotation("Initiating Flush");
+```
+
+Some client operations may have a high volume within your
+application. As such, you may wish to only sample a percentage of
+operations for tracing. As seen below, the CountSampler can be used to
+help enable tracing for 1-in-1000 operations
+
+```java
+import org.apache.htrace.impl.CountSampler;
+...
+Sampler sampler = new CountSampler(HTraceConfiguration.fromMap(
+    Collections.singletonMap(CountSampler.SAMPLER_FREQUENCY_CONF_KEY, "1000")));
+...
+TraceScope readScope = Trace.startSpan("Read", sampler);
+...
+readScope.close();
+```
+
+Remember to close all spans and disable tracing when finished.
+
+```java
+DistributedTrace.disable();
+```
+
+### Viewing Collected Traces
+
+To view collected traces, use the "Recent Traces" link on the Monitor
+UI. You can also programmatically access and print traces using the
+`TraceDump` class.
+
+#### Trace Table Format
+
+This section is for developers looking to use data recorded in the trace table
+directly, above and beyond the default services of the Accumulo monitor.
+Please note the trace table format and its supporting classes
+are not in the public API and may be subject to change in future versions.
+
+Each span received by a tracer's ZooTraceClient is recorded in the trace table
+in the form of three entries: span entries, index entries, and start time entries.
+Span and start time entries record full span information,
+whereas index entries provide indexing into span information
+useful for quickly finding spans by type or start time.
+
+Each entry is illustrated by a description and sample of data.
+In the description, a token in quotes is a String literal,
+whereas other other tokens are span variables.
+Parentheses group parts together, to distinguish colon characters inside the
+column family or qualifier from the colon that separates column family and qualifier.
+We use the format `row columnFamily:columnQualifier columnVisibility    value`
+(omitting timestamp which records the time an entry is written to the trace table).
+
+Span entries take the following form:
+
+    traceId        "span":(parentSpanId:spanId)            []    spanBinaryEncoding
+    63b318de80de96d1 span:4b8f66077df89de1:3778c6739afe4e1 []    %18;%09;...
+
+The parentSpanId is "" for the root span of a trace.
+The spanBinaryEncoding is a compact Apache Thrift encoding of the original Span object.
+This allows clients (and the Accumulo monitor) to recover all the details of the original Span
+at a later time, by scanning the trace table and decoding the value of span entries
+via `TraceFormatter.getRemoteSpan(entry)`.
+
+The trace table has a formatter class by default (org.apache.accumulo.tracer.TraceFormatter)
+that changes how span entries appear from the Accumulo shell.
+Normal scans to the trace table do not use this formatter representation;
+it exists only to make span entries easier to view inside the Accumulo shell.
+
+Index entries take the following form:
+
+    "idx":service:startTime description:sender  []    traceId:elapsedTime
+    idx:tserver:14f3828f58b startScan:localhost []    63b318de80de96d1:1
+
+The service and sender are set by the first call of each Accumulo process
+(and instrumented client processes) to `DistributedTrace.enable(...)`
+(the sender is autodetected if not specified).
+The description is specified in each span.
+Start time and the elapsed time (start - stop, 1 millisecond in the example above)
+are recorded in milliseconds as long values serialized to a string in hex.
+
+Start time entries take the following form:
+
+    "start":startTime "id":traceId        []    spanBinaryEncoding
+    start:14f3828a351 id:63b318de80de96d1 []    %18;%09;...
+
+The following classes may be run while Accumulo is running to provide insight into trace statistics. These require
+accumulo-trace-VERSION.jar to be provided on the Accumulo classpath (`lib/ext` is fine).
+
+    $ accumulo org.apache.accumulo.tracer.TraceTableStats -u username -p password -i instancename
+    $ accumulo org.apache.accumulo.tracer.TraceDump -u username -p password -i instancename -r
+
+### Tracing from the Shell
+You can enable tracing for operations run from the shell by using the
+`trace on` and `trace off` commands.
+
+```
+root@test test> trace on
+
+root@test test> scan
+a b:c []    d
+
+root@test test> trace off
+Waiting for trace information
+Waiting for trace information
+Trace started at 2013/08/26 13:24:08.332
+Time  Start  Service@Location       Name
+ 3628+0      shell@localhost shell:root
+    8+1690     shell@localhost scan
+    7+1691       shell@localhost scan:location
+    6+1692         tserver@localhost startScan
+    5+1692           tserver@localhost tablet read ahead 6
+```
+
+## Logging
+
+Accumulo processes each write to a set of log files. By default, these logs are found at directory
+set by `ACCUMULO_LOG_DIR` in `accumulo-env.sh`.
+
+## Recovery
+
+In the event of TabletServer failure or error on shutting Accumulo down, some
+mutations may not have been minor compacted to HDFS properly. In this case,
+Accumulo will automatically reapply such mutations from the write-ahead log
+either when the tablets from the failed server are reassigned by the Master (in the
+case of a single TabletServer failure) or the next time Accumulo starts (in the event of
+failure during shutdown).
+
+Recovery is performed by asking a tablet server to sort the logs so that tablets can easily find their missing
+updates. The sort status of each file is displayed on
+Accumulo monitor status page. Once the recovery is complete any
+tablets involved should return to an ``online'' state. Until then those tablets will be
+unavailable to clients.
+
+The Accumulo client library is configured to retry failed mutations and in many
+cases clients will be able to continue processing after the recovery process without
+throwing an exception.
+
+## Migrating Accumulo from non-HA Namenode to HA Namenode
+
+The following steps will allow a non-HA instance to be migrated to an HA instance. Consider an HDFS URL
+`hdfs://namenode.example.com:8020` which is going to be moved to `hdfs://nameservice1`.
+
+Before moving HDFS over to the HA namenode, use `accumulo admin volumes` to confirm
+that the only volume displayed is the volume from the current namenode's HDFS URL.
+
+    Listing volumes referenced in zookeeper
+            Volume : hdfs://namenode.example.com:8020/accumulo
+
+    Listing volumes referenced in accumulo.root tablets section
+            Volume : hdfs://namenode.example.com:8020/accumulo
+    Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
+
+    Listing volumes referenced in accumulo.metadata tablets section
+            Volume : hdfs://namenode.example.com:8020/accumulo
+
+    Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
+
+After verifying the current volume is correct, shut down the cluster and transition HDFS to the HA nameservice.
+
+Edit `accumulo-site.xml` to notify accumulo that a volume is being replaced. First,
+add the new nameservice volume to the `instance.volumes` property. Next, add the
+`instance.volumes.replacements` property in the form of `old new`. It's important to not include
+the volume that's being replaced in `instance.volumes`, otherwise it's possible accumulo could continue
+to write to the volume.
+
+```xml
+<!-- instance.dfs.uri and instance.dfs.dir should not be set-->
+<property>
+  <name>instance.volumes</name>
+  <value>hdfs://nameservice1/accumulo</value>
+</property>
+<property>
+  <name>instance.volumes.replacements</name>
+  <value>hdfs://namenode.example.com:8020/accumulo hdfs://nameservice1/accumulo</value>
+</property>
+```
+
+Run `accumulo init --add-volumes` and start up the accumulo cluster. Verify that the
+new nameservice volume shows up with `accumulo admin volumes`.
+
+    Listing volumes referenced in zookeeper
+            Volume : hdfs://namenode.example.com:8020/accumulo
+            Volume : hdfs://nameservice1/accumulo
+
+    Listing volumes referenced in accumulo.root tablets section
+            Volume : hdfs://namenode.example.com:8020/accumulo
+            Volume : hdfs://nameservice1/accumulo
+    Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
+
+    Listing volumes referenced in accumulo.metadata tablets section
+            Volume : hdfs://namenode.example.com:8020/accumulo
+            Volume : hdfs://nameservice1/accumulo
+    Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
+
+Some erroneous GarbageCollector messages may still be seen for a small period while data is transitioning to
+the new volumes. This is expected and can usually be ignored.
+
+## Achieving Stability in a VM Environment
+
+For testing, demonstration, and even operation uses, Accumulo is often
+installed and run in a virtual machine (VM) environment. The majority of
+long-term operational uses of Accumulo are on bare-metal cluster. However, the
+core design of Accumulo and its dependencies do not preclude running stably for
+long periods within a VM. Many of Accumulo’s operational robustness features to
+handle failures like periodic network partitioning in a large cluster carry
+over well to VM environments. This guide covers general recommendations for
+maximizing stability in a VM environment, including some of the common failure
+modes that are more common when running in VMs.
+
+### Known failure modes: Setup and Troubleshooting
+
+In addition to the general failure modes of running Accumulo, VMs can introduce a
+couple of environmental challenges that can affect process stability. Clock
+drift is something that is more common in VMs, especially when VMs are
+suspended and resumed. Clock drift can cause Accumulo servers to assume that
+they have lost connectivity to the other Accumulo processes and/or lose their
+locks in Zookeeper. VM environments also frequently have constrained resources,
+such as CPU, RAM, network, and disk throughput and capacity. Accumulo generally
+deals well with constrained resources from a stability perspective (optimizing
+performance will require additional tuning, which is not covered in this
+section), however there are some limits.
+
+#### Physical Memory
+
+One of those limits has to do with the Linux out of memory killer. A common
+failure mode in VM environments (and in some bare metal installations) is when
+the Linux out of memory killer decides to kill processes in order to avoid a
+kernel panic when provisioning a memory page. This often happens in VMs due to
+the large number of processes that must run in a small memory footprint. In
+addition to the Linux core processes, a single-node Accumulo setup requires a
+Hadoop Namenode, a Hadoop Secondary Namenode a Hadoop Datanode, a Zookeeper
+server, an Accumulo Master, an Accumulo GC and an Accumulo TabletServer.
+Typical setups also include an Accumulo Monitor, an Accumulo Tracer, a Hadoop
+ResourceManager, a Hadoop NodeManager, provisioning software, and client
+applications. Between all of these processes, it is not uncommon to
+over-subscribe the available RAM in a VM. We recommend setting up VMs without
+swap enabled, so rather than performance grinding to a halt when physical
+memory is exhausted the kernel will randomly* select processes to kill in order
+to free up memory.
+
+Calculating the maximum possible memory usage is essential in creating a stable
+Accumulo VM setup. Safely engineering memory allocation for stability is a
+matter of then bringing the calculated maximum memory usage under the physical
+memory by a healthy margin. The margin is to account for operating system-level
+operations, such as managing process, maintaining virtual memory pages, and
+file system caching. When the java out-of-memory killer finds your process, you
+will probably only see evidence of that in /var/log/messages. Out-of-memory
+process kills do not show up in Accumulo or Hadoop logs.
+
+To calculate the max memory usage of all java virtual machine (JVM) processes
+add the maximum heap size (often limited by a -Xmx... argument, such as in
+accumulo-site.xml) and the off-heap memory usage. Off-heap memory usage
+includes the following:
+
+* "Permanent Space", where the JVM stores Classes, Methods, and other code elements. This can be limited by a JVM flag such as `-XX:MaxPermSize:100m`, and is typically tens of megabytes.
+* Code generation space, where the JVM stores just-in-time compiled code. This is typically small enough to ignore
+* Socket buffers, where the JVM stores send and receive buffers for each socket.
+* Thread stacks, where the JVM allocates memory to manage each thread.
+* Direct memory space and JNI code, where applications can allocate memory outside of the JVM-managed space. For Accumulo, this includes the native in-memory maps that are allocated with the memory.maps.max parameter in accumulo-site.xml.
+* Garbage collection space, where the JVM stores information used for garbage collection.
+
+You can assume that each Hadoop and Accumulo process will use ~100-150MB for
+Off-heap memory, plus the in-memory map of the Accumulo TServer process. A
+simple calculation for physical memory requirements follows:
+
+```
+  Physical memory needed
+    = (per-process off-heap memory) + (heap memory) + (other processes) + (margin)
+    = (number of java processes * 150M + native map) + (sum of -Xmx settings for java process) + (total applications memory, provisioning memory, etc.) + (1G)
+    = (11*150M +500M) + (1G +1G +1G +256M +1G +256M +512M +512M +512M +512M +512M) + (2G) + (1G)
+    = (2150M) + (7G) + (2G) + (1G)
+    = ~12GB
+```
+
+These calculations can add up quickly with the large number of processes,
+especially in constrained VM environments. To reduce the physical memory
+requirements, it is a good idea to reduce maximum heap limits and turn off
+unnecessary processes. If you're not using YARN in your application, you can
+turn off the ResourceManager and NodeManager. If you're not expecting to
+re-provision the cluster frequently you can turn off or reduce provisioning
+processes such as Salt Stack minions and masters.
+
+#### Disk Space
+
+Disk space is primarily used for two operations: storing data and storing logs.
+While Accumulo generally stores all of its key/value data in HDFS, Accumulo,
+Hadoop, and Zookeeper all store a significant amount of logs in a directory on
+a local file system. Care should be taken to make sure that (a) limitations to
+the amount of logs generated are in place, and (b) enough space is available to
+host the generated logs on the partitions that they are assigned. When space is
+not available to log, processes will hang. This can cause interruptions in
+availability of Accumulo, as well as cascade into failures of various
+processes.
+
+Hadoop, Accumulo, and Zookeeper use log4j as a logging mechanism, and each of
+them has a way of limiting the logs and directing them to a particular
+directory. Logs are generated independently for each process, so when
+considering the total space you need to add up the maximum logs generated by
+each process. Typically, a rolling log setup in which each process can generate
+something like 10 100MB files is instituted, resulting in a maximum file system
+usage of 1GB per process. Default setups for Hadoop and Zookeeper are often
+unbounded, so it is important to set these limits in the logging configuration
+files for each subsystem. Consult the user manual for each system for
+instructions on how to limit generated logs.
+
+#### Zookeeper Interaction
+
+Accumulo is designed to scale up to thousands of nodes. At that scale,
+intermittent interruptions in network service and other rare failures of
+compute nodes become more common. To limit the impact of node failures on
+overall service availability, Accumulo uses a heartbeat monitoring system that
+leverages Zookeeper's ephemeral locks. There are several conditions that can
+occur that cause Accumulo process to lose their Zookeeper locks, some of which
+are true interruptions to availability and some of which are false positives.
+Several of these conditions become more common in VM environments, where they
+can be exacerbated by resource constraints and clock drift.
+
+#### Tested Versions
+
+Each release of Accumulo is built with a specific version of Apache
+Hadoop, Apache ZooKeeper and Apache Thrift.  We expect Accumulo to
+work with versions that are API compatible with those versions.
+However this compatibility is not guaranteed because Hadoop, ZooKeeper
+and Thift may not provide guarantees between their own versions. We
+have also found that certain versions of Accumulo and Hadoop included
+bugs that greatly affected overall stability.  Thrift is particularly
+prone to compatibility changes between versions and you must use the
+same version your Accumulo is built with.
+
+Please check the release notes for your Accumulo version or use the
+mailing lists at https://accumulo.apache.org for more info.
+
+[tracing]: {{page.docs_baseurl}}/administration/overview#tracing
+[monitor]: {{page.docs_baseurl}}/administration/overview#monitoring
+[config-mgmt]: {{page.docs_baseurl}}/administration/configuration-management
+[config-props]: {{page.docs_baseurl}}/administration/configuration-properties

http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/7cc70b2e/_docs-unreleased/administration/replication.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/administration/replication.md b/_docs-unreleased/administration/replication.md
new file mode 100644
index 0000000..edfa1f8
--- /dev/null
+++ b/_docs-unreleased/administration/replication.md
@@ -0,0 +1,385 @@
+---
+title: Replication
+category: administration
+order: 5
+---
+
+## Overview
+
+Replication is a feature of Accumulo which provides a mechanism to automatically
+copy data to other systems, typically for the purpose of disaster recovery,
+high availability, or geographic locality. It is best to consider this feature
+as a framework for automatic replication instead of the ability to copy data
+from to another Accumulo instance as copying to another Accumulo cluster is
+only an implementation detail. The local Accumulo cluster is hereby referred
+to as the `primary` while systems being replicated to are known as
+`peers`.
+
+This replication framework makes two Accumulo instances, where one instance
+replicates to another, eventually consistent between one another, as opposed
+to the strong consistency that each single Accumulo instance still holds. That
+is to say, attempts to read data from a table on a peer which has pending replication
+from the primary will not wait for that data to be replicated before running the scan.
+This is desirable for a number of reasons, the most important is that the replication
+framework is not limited by network outages or offline peers, but only by the HDFS
+space available on the primary system.
+
+Replication configurations can be considered as a directed graph which allows cycles.
+The systems in which data was replicated from is maintained in each Mutation which
+allow each system to determine if a peer has already has the data in which
+the system wants to send.
+
+Data is replicated by using the Write-Ahead logs (WAL) that each TabletServer is
+already maintaining. TabletServers records which WALs have data that need to be
+replicated to the `accumulo.metadata` table. The Master uses these records,
+combined with the local Accumulo table that the WAL was used with, to create records
+in the `replication` table which track which peers the given WAL should be
+replicated to. The Master latter uses these work entries to assign the actual
+replication task to a local TabletServer using ZooKeeper. A TabletServer will get
+a lock in ZooKeeper for the replication of this file to a peer, and proceed to
+replicate to the peer, recording progress in the `replication` table as
+data is successfully replicated on the peer. Later, the Master and Garbage Collector
+will remove records from the `accumulo.metadata` and `replication` tables
+and files from HDFS, respectively, after replication to all peers is complete.
+
+## Configuration
+
+Configuration of Accumulo to replicate data to another system can be categorized
+into the following sections.
+
+### Site Configuration
+
+Each system involved in replication (even the primary) needs a name that uniquely
+identifies it across all peers in the replication graph. This should be considered
+fixed for an instance, and set in `accumulo-site.xml`.
+
+```xml
+<property>
+    <name>replication.name</name>
+    <value>primary</value>
+    <description>Unique name for this system used by replication</description>
+</property>
+```
+
+### Instance Configuration
+
+For each peer of this system, Accumulo needs to know the name of that peer,
+the class used to replicate data to that system and some configuration information
+to connect to this remote peer. In the case of Accumulo, this additional data
+is the Accumulo instance name and ZooKeeper quorum; however, this varies on the
+replication implementation for the peer.
+
+These can be set in the site configuration to ease deployments; however, as they may
+change, it can be useful to set this information using the Accumulo shell.
+
+To configure a peer with the name `peer1` which is an Accumulo system with an instance name of `accumulo_peer`
+and a ZooKeeper quorum of `10.0.0.1,10.0.2.1,10.0.3.1`, invoke the following
+command in the shell.
+
+```
+root@accumulo_primary> config -s
+replication.peer.peer1=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,accumulo_peer,10.0.0.1,10.0.2.1,10.0.3.1
+```
+
+Since this is an Accumulo system, we also want to set a username and password
+to use when authenticating with this peer. On our peer, we make a special user
+which has permission to write to the tables we want to replicate data into, "replication"
+with a password of "password". We then need to record this in the primary's configuration.
+
+```
+root@accumulo_primary> config -s replication.peer.user.peer1=replication
+root@accumulo_primary> config -s replication.peer.password.peer1=password
+```
+
+Alternatively, when configuring replication on Accumulo running Kerberos, a keytab
+file per peer can be configured instead of a password. The provided keytabs must be readable
+by the unix user running Accumulo. They keytab for a peer can be unique from the
+keytab used by Accumulo or any keytabs for other peers.
+
+```
+accumulo@EXAMPLE.COM@accumulo_primary> config -s replication.peer.user.peer1=replication@EXAMPLE.COM
+accumulo@EXAMPLE.COM@accumulo_primary> config -s replication.peer.keytab.peer1=/path/to/replication.keytab
+```
+
+### Table Configuration
+
+Now, we presently have a peer defined, so we just need to configure which tables will
+replicate to that peer. We also need to configure an identifier to determine where
+this data will be replicated on the peer. Since we're replicating to another Accumulo
+cluster, this is a table ID. In this example, we want to enable replication on
+`my_table` and configure our peer `accumulo_peer` as a target, sending
+the data to the table with an ID of `2` in `accumulo_peer`.
+
+```
+root@accumulo_primary> config -t my_table -s table.replication=true
+root@accumulo_primary> config -t my_table -s table.replication.target.accumulo_peer=2
+```
+
+To replicate a single table on the primary to multiple peers, the second command
+in the above shell snippet can be issued, for each peer and remote identifier pair.
+
+## Monitoring
+
+Basic information about replication status from a primary can be found on the Accumulo
+Monitor server, using the `Replication` link the sidebar.
+
+On this page, information is broken down into the following sections:
+
+1. Files pending replication by peer and target
+2. Files queued for replication, with progress made
+
+## Work Assignment
+
+Depending on the schema of a table, different implementations of the WorkAssigner used could
+be configured. The implementation is controlled via the property `replication.work.assigner`
+and the full class name for the implementation. This can be configured via the shell or
+`accumulo-site.xml`.
+
+```xml
+<property>
+    <name>replication.work.assigner</name>
+    <value>org.apache.accumulo.master.replication.SequentialWorkAssigner</value>
+    <description>Implementation used to assign work for replication</description>
+</property>
+```
+
+```
+root@accumulo_primary> config -t my_table -s replication.work.assigner=org.apache.accumulo.master.replication.SequentialWorkAssigner
+```
+
+Two implementations are provided. By default, the `SequentialWorkAssigner` is configured for an
+instance. The SequentialWorkAssigner ensures that, per peer and each remote identifier, each WAL is
+replicated in the order in which they were created. This is sufficient to ensure that updates to a table
+will be replayed in the correct order on the peer. This implementation has the downside of only replicating
+a single WAL at a time.
+
+The second implementation, the `UnorderedWorkAssigner` can be used to overcome the limitation
+of only a single WAL being replicated to a target and peer at any time. Depending on the table schema,
+it's possible that multiple versions of the same Key with different values are infrequent or nonexistent.
+In this case, parallel replication to a peer and target is possible without any downsides. In the case
+where this implementation is used were column updates are frequent, it is possible that there will be
+an inconsistency between the primary and the peer.
+
+## ReplicaSystems
+
+`ReplicaSystem` is the interface which allows abstraction of replication of data
+to peers of various types. Presently, only an `AccumuloReplicaSystem` is provided
+which will replicate data to another Accumulo instance. A `ReplicaSystem` implementation
+is run inside of the TabletServer process, and can be configured as mentioned in the 
+`Instance Configuration` section of this document. Theoretically, an implementation
+of this interface could send data to other filesystems, databases, etc.
+
+### AccumuloReplicaSystem
+
+The `AccumuloReplicaSystem` uses Thrift to communicate with a peer Accumulo instance
+and replicate the necessary data. The TabletServer running on the primary will communicate
+with the Master on the peer to request the address of a TabletServer on the peer which
+this TabletServer will use to replicate the data.
+
+The TabletServer on the primary will then replicate data in batches of a configurable
+size (`replication.max.unit.size`). The TabletServer on the peer will report how many
+records were applied back to the primary, which will be used to record how many records
+were successfully replicated. The TabletServer on the primary will continue to replicate
+data in these batches until no more data can be read from the file.
+
+## Other Configuration
+
+There are a number of configuration values that can be used to control how
+the implementation of various components operate.
+
+|Property | Description | Default
+|---------|-------------|--------
+|replication.max.work.queue | Maximum number of files queued for replication at one time | 1000
+|replication.work.assignment.sleep | Time between invocations of the WorkAssigner | 30s
+|replication.worker.threads | Size of threadpool used to replicate data to peers | 4
+|replication.receipt.service.port | Thrift service port to listen for replication requests, can use '0' for a random port | 10002
+|replication.work.attempts | Number of attempts to replicate to a peer before aborting the attempt | 10
+|replication.receiver.min.threads | Minimum number of idle threads for handling incoming replication | 1
+|replication.receiver.threadcheck.time | Time between attempting adjustments of thread pool for incoming replications | 30s
+|replication.max.unit.size | Maximum amount of data to be replicated in one RPC | 64M
+|replication.work.assigner | Work Assigner implementation | org.apache.accumulo.master.replication.SequentialWorkAssigner
+|tserver.replication.batchwriter.replayer.memory| Size of BatchWriter cache to use in applying replication requests | 50M
+
+## Example Practical Configuration
+
+A real-life example is now provided to give concrete application of replication configuration. This
+example is a two instance Accumulo system, one primary system and one peer system. They are called
+primary and peer, respectively. Each system also have a table of the same name, "my_table". The instance
+name for each is also the same (primary and peer), and both have ZooKeeper hosts on a node with a hostname
+with that name as well (primary:2181 and peer:2181).
+
+We want to configure these systems so that "my_table" on "primary" replicates to "my_table" on "peer".
+
+### accumulo-site.xml
+
+We can assign the "unique" name that identifies this Accumulo instance among all others that might participate
+in replication together. In this example, we will use the names provided in the description.
+
+#### Primary
+
+```xml
+<property>
+  <name>replication.name</name>
+  <value>primary</value>
+  <description>Defines the unique name</description>
+</property>
+```
+
+#### Peer
+
+```xml
+<property>
+  <name>replication.name</name>
+  <value>peer</value>
+</property>
+```
+
+### masters and tservers files
+
+Be *sure* to use non-local IP addresses. Other nodes need to connect to it and using localhost will likely result in
+a local node talking to another local node.
+
+### Start both instances
+
+The rest of the configuration is dynamic and is best configured on the fly (in ZooKeeper) than in accumulo-site.xml.
+
+### Peer
+
+The next series of command are to be run on the peer system. Create a user account for the primary instance called
+"peer". The password for this account will need to be saved in the configuration on the primary
+
+```
+root@peer> createtable my_table
+root@peer> createuser peer
+root@peer> grant -t my_table -u peer Table.WRITE
+root@peer> grant -t my_table -u peer Table.READ
+root@peer> tables -l
+```
+
+Remember what the table ID for 'my_table' is. You'll need that to configured the primary instance.
+
+### Primary
+
+Next, configure the primary instance.
+
+#### Set up the table
+
+```
+root@primary> createtable my_table
+```
+
+#### Define the Peer as a replication peer to the Primary
+
+We're defining the instance with replication.name of 'peer' as a peer. We provide the implementation of ReplicaSystem
+that we want to use, and the configuration for the AccumuloReplicaSystem. In this case, the configuration is the Accumulo
+Instance name for 'peer' and the ZooKeeper quorum string. The configuration key is of the form
+"replication.peer.$peer_name".
+
+```
+root@primary> config -s replication.peer.peer=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,peer,$peer_zk_quorum
+```
+
+#### Set the authentication credentials
+
+We want to use that special username and password that we created on the peer, so we have a means to write data to
+the table that we want to replicate to. The configuration key is of the form "replication.peer.user.$peer_name".
+
+```
+root@primary> config -s replication.peer.user.peer=peer
+root@primary> config -s replication.peer.password.peer=peer
+```
+
+#### Enable replication on the table
+
+Now that we have defined the peer on the primary and provided the authentication credentials, we need to configure
+our table with the implementation of ReplicaSystem we want to use to replicate to the peer. In this case, our peer 
+is an Accumulo instance, so we want to use the AccumuloReplicaSystem.
+
+The configuration for the AccumuloReplicaSystem is the table ID for the table on the peer instance that we
+want to replicate into. Be sure to use the correct value for $peer_table_id. The configuration key is of
+the form "table.replication.target.$peer_name".
+
+```
+root@primary> config -t my_table -s table.replication.target.peer=$peer_table_id
+```
+
+Finally, we can enable replication on this table.
+
+```
+root@primary> config -t my_table -s table.replication=true
+```
+
+## Extra considerations for use
+
+While this feature is intended for general-purpose use, its implementation does carry some baggage. Like any software,
+replication is a feature that operates well within some set of use cases but is not meant to support all use cases.
+For the benefit of the users, we can enumerate these cases.
+
+### Latency
+
+As previously mentioned, the replication feature uses the Write-Ahead Log files for a number of reasons, one of which
+is to prevent the need for data to be written to RFiles before it is available to be replicated. While this can help
+reduce the latency for a batch of Mutations that have been written to Accumulo, the latency is at least seconds to tens
+of seconds for replication once ingest is active. For a table which replication has just been enabled on, this is likely
+to take a few minutes before replication will begin.
+
+Once ingest is active and flowing into the system at a regular rate, replication should be occurring at a similar rate, 
+given sufficient computing resources. Replication attempts to copy data at a rate that is to be considered low latency
+but is not a replacement for custom indexing code which can ensure near real-time referential integrity on secondary indexes.
+
+### Table-Configured Iterators
+
+Accumulo Iterators tend to be a heavy hammer which can be used to solve a variety of problems. In general, it is highly
+recommended that Iterators which are applied at major compaction time are both idempotent and associative due to the
+non-determinism in which some set of files for a Tablet might be compacted. In practice, this translates to common patterns,
+such as aggregation, which are implemented in a manner resilient to duplication (such as using a Set instead of a List).
+
+Due to the asynchronous nature of replication and the expectation that hardware failures and network partitions will exist,
+it is generally not recommended to not configure replication on a table which has Iterators set which are not idempotent.
+While the replication implementation can make some simple assertions to try to avoid re-replication of data, it is not
+presently guaranteed that all data will only be sent to a peer once. Data will be replicated at least once. Typically,
+this is not a problem as the VersioningIterator will automaticaly deduplicate this over-replication because they will
+have the same timestamp; however, certain Combiners may result in inaccurate aggregations.
+
+As a concrete example, consider a table which has the SummingCombiner configured to sum all values for
+multiple versions of the same Key. For some key, consider a set of numeric values that are written to a table on the
+primary: [1, 2, 3]. On the primary, all of these are successfully written and thus the current value for the given key
+would be 6, (1 + 2 + 3). Consider, however, that each of these updates to the peer were done independently (because
+other data was also included in the write-ahead log that needed to be replicated). The update with a value of 1 was
+successfully replicated, and then we attempted to replicate the update with a value of 2 but the remote server never
+responded. The primary does not know whether the update with a value of 2 was actually applied or not, so the
+only recourse is to re-send the update. After we receive confirmation that the update with a value of 2 was replicated,
+we will then replicate the update with 3. If the peer did never apply the first update of '2', the summation is accurate.
+If the update was applied but the acknowledgement was lost for some reason (system failure, network partition), the
+update will be resent to the peer. Because addition is non-idempotent, we have created an inconsistency between the
+primary and peer. As such, the SummingCombiner wouldn't be recommended on a table being replicated.
+
+While there are changes that could be made to the replication implementation which could attempt to mitigate this risk,
+presently, it is not recommended to configure Iterators or Combiners which are not idempotent to support cases where
+inaccuracy of aggregations is not acceptable.
+
+### Duplicate Keys
+
+In Accumulo, when more than one key exists that are exactly the same, keys that are equal down to the timestamp,
+the retained value is non-deterministic. Replication introduces another level of non-determinism in this case.
+For a table that is being replicated and has multiple equal keys with different values inserted into it, the final
+value in that table on the primary instance is not guaranteed to be the final value on all replicas.
+
+For example, say the values that were inserted on the primary instance were `value1` and `value2` and the final
+value was `value1`, it is not guaranteed that all replicas will have `value1` like the primary. The final value is
+non-deterministic for each instance.
+
+As is the recommendation without replication enabled, if multiple values for the same key (sans timestamp) are written to
+Accumulo, it is strongly recommended that the value in the timestamp properly reflects the intended version by
+the client. That is to say, newer values inserted into the table should have larger timestamps. If the time between
+writing updates to the same key is significant (order minutes), this concern can likely be ignored.
+
+### Bulk Imports
+
+Currently, files that are bulk imported into a table configured for replication are not replicated. There is no
+technical reason why it was not implemented, it was simply omitted from the initial implementation. This is considered a
+fair limitation because bulk importing generated files multiple locations is much simpler than bifurcating "live" ingest
+data into two instances. Given some existing bulk import process which creates files and them imports them into an
+Accumulo instance, it is trivial to copy those files to a new HDFS instance and import them into another Accumulo
+instance using the same process. Hadoop's `distcp` command provides an easy way to copy large amounts of data to another
+HDFS instance which makes the problem of duplicating bulk imports very easy to solve.

http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/7cc70b2e/_docs-unreleased/administration/ssl.md
----------------------------------------------------------------------
diff --git a/_docs-unreleased/administration/ssl.md b/_docs-unreleased/administration/ssl.md
new file mode 100644
index 0000000..a104adf
--- /dev/null
+++ b/_docs-unreleased/administration/ssl.md
@@ -0,0 +1,124 @@
+---
+title: SSL
+category: administration
+order: 8
+---
+
+Accumulo, through Thrift's TSSLTransport, provides the ability to encrypt
+wire communication between Accumulo servers and clients using secure
+sockets layer (SSL). SSL certifcates signed by the same certificate authority
+control the "circle of trust" in which a secure connection can be established.
+Typically, each host running Accumulo processes would be given a certificate
+which identifies itself.
+
+Clients can optionally also be given a certificate, when client-auth is enabled,
+which prevents unwanted clients from accessing the system. The SSL integration
+presently provides no authentication support within Accumulo (an Accumulo username
+and password are still required) and is only used to establish a means for
+secure communication.
+
+## Server configuration
+
+As previously mentioned, the circle of trust is established by the certificate
+authority which created the certificates in use. Because of the tight coupling
+of certificate generation with an organization's policies, Accumulo does not
+provide a method in which to automatically create the necessary SSL components.
+
+Administrators without existing infrastructure built on SSL are encourage to
+use OpenSSL and the `keytool` command. An example of these commands are
+included in a section below. Accumulo servers require a certificate and keystore,
+in the form of Java KeyStores, to enable SSL. The following configuration assumes
+these files already exist.
+
+In `accumulo-site.xml`, the following properties are required:
+
+* **rpc.javax.net.ssl.keyStore**=_The path on the local filesystem to the keystore containing the server's certificate_
+* **rpc.javax.net.ssl.keyStorePassword**=_The password for the keystore containing the server's certificate_
+* **rpc.javax.net.ssl.trustStore**=_The path on the local filesystem to the keystore containing the certificate authority's public key_
+* **rpc.javax.net.ssl.trustStorePassword**=_The password for the keystore containing the certificate authority's public key_
+* **instance.rpc.ssl.enabled**=_true_
+
+Optionally, SSL client-authentication (two-way SSL) can also be enabled by setting
+`instance.rpc.ssl.clientAuth=true` in `accumulo-site.xml`.
+This requires that each client has access to  valid certificate to set up a secure connection
+to the servers. By default, Accumulo uses one-way SSL which does not require clients to have
+their own certificate.
+
+## Client configuration
+
+To establish a connection to Accumulo servers, each client must also have
+special configuration. This is typically accomplished through the use of
+the client configuration file whose default location is `~/.accumulo/config`.
+
+The following properties must be set to connect to an Accumulo instance using SSL:
+
+* **rpc.javax.net.ssl.trustStore**=_The path on the local filesystem to the keystore containing the certificate authority's public key_
+* **rpc.javax.net.ssl.trustStorePassword**=_The password for the keystore containing the certificate authority's public key_
+* **instance.rpc.ssl.enabled**=_true_
+
+If two-way SSL if enabled (`instance.rpc.ssl.clientAuth=true`) for the instance, the client must also define
+their own certificate and enable client authenticate as well.
+
+* **rpc.javax.net.ssl.keyStore**=_The path on the local filesystem to the keystore containing the server's certificate_
+* **rpc.javax.net.ssl.keyStorePassword**=_The password for the keystore containing the server's certificate_
+* **instance.rpc.ssl.clientAuth**=_true_
+
+## Generating SSL material using OpenSSL
+
+The following is included as an example for generating your own SSL material (certificate authority and server/client
+certificates) using OpenSSL and Java's KeyTool command.
+
+### Generate a certificate authority
+
+```shell
+# Create a private key
+openssl genrsa -des3 -out root.key 4096
+
+# Create a certificate request using the private key
+openssl req -x509 -new -key root.key -days 365 -out root.pem
+
+# Generate a Base64-encoded version of the PEM just created
+openssl x509 -outform der -in root.pem -out root.der
+
+# Import the key into a Java KeyStore
+keytool -import -alias root-key -keystore truststore.jks -file root.der
+
+# Remove the DER formatted key file (as we don't need it anymore)
+rm root.der
+```
+
+The `truststore.jks` file is the Java keystore which contains the certificate authority's public key.
+
+### Generate a certificate/keystore per host
+
+It's common that each host in the instance is issued its own certificate (notably to ensure that revocation procedures
+can be easily followed). The following steps can be taken for each host.
+
+```shell
+# Create the private key for our server
+openssl genrsa -out server.key 4096
+
+# Generate a certificate signing request (CSR) with our private key
+openssl req -new -key server.key -out server.csr
+
+# Use the CSR and the CA to create a certificate for the server (a reply to the CSR)
+openssl x509 -req -in server.csr -CA root.pem -CAkey root.key -CAcreateserial \
+    -out server.crt -days 365
+
+# Use the certificate and the private key for our server to create PKCS12 file
+openssl pkcs12 -export -in server.crt -inkey server.key -certfile server.crt \
+    -name 'server-key' -out server.p12
+
+# Create a Java KeyStore for the server using the PKCS12 file (private key)
+keytool -importkeystore -srckeystore server.p12 -srcstoretype pkcs12 -destkeystore \
+    server.jks -deststoretype JKS
+
+# Remove the PKCS12 file as we don't need it
+rm server.p12
+
+# Import the CA-signed certificate to the keystore
+keytool -import -trustcacerts -alias server-crt -file server.crt -keystore server.jks
+```
+
+The `server.jks` file is the Java keystore containing the certificate for a given host. The above
+methods are equivalent whether the certficate is generate for an Accumulo server or a client.


Mime
View raw message