Mailing-List: contact commits-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: elserj@apache.org
To: commits@accumulo.apache.org
Date: Fri, 19 Dec 2014 18:44:44 -0000
Message-Id: <04b278ea61894e739d4ea790e795217b@git.apache.org>
In-Reply-To: <5ba5da39cf36487c985e964c4efce06d@git.apache.org>
References: <5ba5da39cf36487c985e964c4efce06d@git.apache.org>
Subject: [3/8] accumulo git commit: ACCUMULO-3436 Add implementation details
 section to manual

ACCUMULO-3436 Add implementation details section to manual

Only subsection of this is FATE: overview on why it exists,
very general details on how it works, and how users/admins can
interact with it via the shell.


Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/c7f45025
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/c7f45025
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/c7f45025

Branch: refs/heads/1.6
Commit: c7f450258bdfcfa248533d1d45f8963c4cd0bdca
Parents: 67998ed
Author: Josh Elser <elserj@apache.org>
Authored: Fri Dec 19 13:23:39 2014 -0500
Committer: Josh Elser <elserj@apache.org>
Committed: Fri Dec 19 13:23:39 2014 -0500

----------------------------------------------------------------------
 .../accumulo_user_manual.tex                    |  1 +
 .../chapters/implementation.tex                 | 76 ++++++++++++++++++++
 2 files changed, 77 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/accumulo/blob/c7f45025/docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex
----------------------------------------------------------------------
diff --git a/docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex b/docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex
index ae2a440..be26a48 100644
--- a/docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex
+++ b/docs/src/main/latex/accumulo_user_manual/accumulo_user_manual.tex
@@ -47,6 +47,7 @@ Version 1.6}
 \include{chapters/high_speed_ingest}
 \include{chapters/analytics}
 \include{chapters/security}
+\include{chapters/implementation}
 \include{chapters/administration}
 \include{chapters/multivolume}
 \include{chapters/troubleshooting}

http://git-wip-us.apache.org/repos/asf/accumulo/blob/c7f45025/docs/src/main/latex/accumulo_user_manual/chapters/implementation.tex
----------------------------------------------------------------------
diff --git a/docs/src/main/latex/accumulo_user_manual/chapters/implementation.tex b/docs/src/main/latex/accumulo_user_manual/chapters/implementation.tex
new file mode 100644
index 0000000..6e7f7bd
--- /dev/null
+++ b/docs/src/main/latex/accumulo_user_manual/chapters/implementation.tex
@@ -0,0 +1,76 @@
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to You under the Apache License, Version 2.0
+% (the "License"); you may not use this file except in compliance with
+% the License. You may obtain a copy of the License at
+%
+%     http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+% See the License for the specific language governing permissions and
+% limitations under the License.
+
+\chapter{Implementation Details}
+
+\section{Fault-Tolerant Executor (FATE)}
+
+Accumulo must implement a number of distributed, multi-step operations to support
+the client API. Creating a new table is a simple example of an atomic client call
+which requires multiple steps in the implementation: get a unique table ID, configure
+default table permissions, populate information in ZooKeeper to record the table's
+existence, create directories in HDFS for the table's data, etc. Implementing these
+steps in a way that is tolerant to node failure and other concurrent operations is
+very difficult to achieve. Accumulo includes a Fault-Tolerant Executor (FATE) which
+is widely used server-side to implement the client API safely and correctly.
+
+FATE is the implementation detail which ensures that tables in creation when the
+Master dies will be successfully created when another Master process is started.
+This alleviates the need for any external tools to correct some bad state -- Accumulo can
+undo the failure and self-heal without any external intervention.
+
+\subsection{Overview}
+
+FATE consists of two primary components: a repeatable, persisted operation (REPO), a storage
+layer for REPOs and an execution system to run REPOs. Accumulo uses ZooKeeper as the storage
+layer for FATE and the Accumulo Master acts as the execution system to run REPOs.
+
+The important characteristic of REPOs are that they implemented in a way that is idempotent:
+every operation must be able to undo or replay a partial execution of itself. Requiring the 
+implementation of the operation to support this functional greatly simplifies the execution
+of these operations. This property is also what guarantees safety in light of failure conditions.
+
+\subsection{Administration}
+
+Sometimes, it is useful to inspect the current FATE operations, both pending and executing.
+For example, a command that is not completing could be blocked on the execution of another
+operation. Accumulo provides an Accumulo shell command to interact with fate.
+
+The \texttt{fate} shell command accepts a number of arguments for different functionality:
+\texttt{list}/\texttt{print}, \texttt{fail}, \texttt{delete}.
+
+\subsubsection{List/Print}
+
+Without any additional arguments, this command will print all operations that still exist in
+the FATE store (ZooKeeper). This will include active, pending, and completed operations (completed
+operations are lazily removed from the store). Each operation includes a unique "transaction ID", the
+state of the operation (e.g. \texttt{NEW}, \texttt{IN\_PROGRESS}, \texttt{FAILED}), any locks the
+transaction actively holds and any locks it is waiting to acquire.
+
+This option can also accept transaction IDs which will restrict the list of transactions shown.
+
+\subsubsection{Fail}
+
+This command can be used to manually fail a FATE transaction and requires a transaction ID
+as an argument. Failing an operation is not a normal procedure and should only be performed
+by an administrator who understands the implications of why they are failing the operation.
+
+\subsubsection{Delete}
+
+This command requires a transaction ID and will delete any locks that the transaction
+holds. Like the fail command, this command should only be used in extreme circumstances
+by an administrator that understands the implications of the command they are about to 
+invoke. It is not normal to invoke this command.