fluo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mwa...@apache.org
Subject [fluo-website] branch gh-pages updated: Fixes #82 Moved docs source from Fluo repo to website (#83)
Date Mon, 25 Sep 2017 20:02:46 GMT
This is an automated email from the ASF dual-hosted git repository.

mwalch pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/fluo-website.git

The following commit(s) were added to refs/heads/gh-pages by this push:
     new 55511fd  Fixes #82 Moved docs source from Fluo repo to website (#83)
55511fd is described below

commit 55511fdbf89da3bc2825d8cf9194608e99198bd0
Author: Mike Walch <mwalch@apache.org>
AuthorDate: Mon Sep 25 16:02:44 2017 -0400

    Fixes #82 Moved docs source from Fluo repo to website (#83)
 _config.yml                                |  18 ++
 _fluo-docs-1-2/administration/metrics.md   | 189 +++++++++++++++++
 _fluo-docs-1-2/development/applications.md | 314 +++++++++++++++++++++++++++++
 _fluo-docs-1-2/getting-started/design.md   |  45 +++++
 _fluo-docs-1-2/getting-started/install.md  | 115 +++++++++++
 _fluo-docs-1-2/index.md                    |   4 +
 _layouts/default.html                      |   8 +-
 _layouts/fluo-docs-1.2.html                |  55 +++++
 css/fluo.scss                              |   6 +
 docs/fluo/1.0.0-incubating/index.md        |   2 +
 docs/fluo/1.1.0-incubating/index.md        |   2 +
 docs/index.md                              |   4 +-
 resources/docs/fluo-architecture.odg       | Bin 0 -> 16670 bytes
 resources/docs/fluo-architecture.png       | Bin 0 -> 61085 bytes
 14 files changed, 758 insertions(+), 4 deletions(-)

diff --git a/_config.yml b/_config.yml
index 423b318..657cb54 100644
--- a/_config.yml
+++ b/_config.yml
@@ -5,6 +5,12 @@ timezone: Etc/UTC
 permalink: pretty
+# Collection names cannot contain periods
+  fluo-docs-1-2:
+    output: true
+    permalink: "/docs/fluo/1.2/:path"
@@ -34,6 +40,18 @@ defaults:
       layout: "tour"
       permalink: "/tour/:basename/"
+  -
+    scope:
+      path: "_fluo-docs-1-2"
+      type: "fluo-docs-1-2"
+    values:
+      layout: "fluo-docs-1.2"
+      title_prefix: "Fluo Documentation - "
+      version: "1.2.0"
+      minor_release: "1.2"
+      docs_base: "/docs/fluo/1.2"
+      javadoc_base: "https://static.javadoc.io/org.apache.fluo/fluo-api/1.1.0-incubating"
+      github_base: "https://github.com/apache/fluo/blob/master"
 # Number of posts displayed on the home page.
 num_home_posts: 5
diff --git a/_fluo-docs-1-2/administration/metrics.md b/_fluo-docs-1-2/administration/metrics.md
new file mode 100644
index 0000000..9576ee6
--- /dev/null
+++ b/_fluo-docs-1-2/administration/metrics.md
@@ -0,0 +1,189 @@
+title: Metrics
+category: administration
+order: 1
+A Fluo application can be configured (in [fluo-app.properties]) to report metrics. When metrics
+configured, Fluo will report some 'default' metrics about an application that help users
monitor its
+performance. Users can also write code to report 'application-specific' metrics from their
+applications. Both 'application-specific' and 'default' metrics share the same reporter configured
+by [fluo-app.properties] and are described in detail below.
+## Configuring reporters
+Fluo metrics are not published by default. To publish metrics, configure a reporter in the
+section of [fluo-app.properties]. There are several different reporter types (i.e Console,
+Graphite, JMX, SLF4J) that are implemented using [Dropwizard]. The choice of which reporter
to use
+depends on the visualization tool used. If you are not currently using a visualization tool,
+is documentation at end of this page for reporting Fluo metrics to Grafana/InfluxDB.
+## Metrics names
+When Fluo metrics are reported, they are published using a naming scheme that encodes additional
+information. This additional information is represented using all caps variables (i.e `METRIC`)
+Default metrics start with `fluo.class` or `fluo.system` and have following naming schemes:
+Application metrics start with `fluo.app` and have following scheme:
+        fluo.app.REPORTER_ID.METRIC
+The variables below describe the additional information that is encoded in metrics names.
+1. `APPLICATION` - Fluo application name
+2. `REPORTER_ID` - Unique ID of the Fluo oracle, worker, or client that is reporting the
+    When running in YARN, this ID is of the format `worker-INSTANCE_ID` or `oracle-INSTANCE_ID`
+    where `INSTANCE_ID` corresponds to instance number. When not running in YARN, this ID
+    of a hostname and a base36 long that is unique across all fluo processes.
+3. `METRIC` - Name of the metric. For 'default' metrics, this is set by Fluo. For 'application'
+    metrics, this is set by user. Name should be unique and avoid using period '.' in name.
+4. `CLASS` - Name of Fluo observer or loader class that produced metric. This allows things
+    transaction collisions to be tracked per class.
+## Application-specific metrics
+Application metrics are implemented by retrieving a [MetricsReporter] from an [Observer],
+or [FluoClient].  These metrics are named using the format `fluo.app.REPORTER_ID.METRIC`.
+## Default metrics
+Default metrics report for a particular Observer/Loader class or system-wide.
+Below are metrics that are reported from each Observer/Loader class that is configured in
a Fluo
+application. These metrics are reported after each transaction and named using the format
+* tx_lock_wait_time - [Timer]
+    - Time transaction spent waiting on locks held by other transactions.
+    - Only updated for transactions that have non-zero lock time.
+* tx_execution_time - [Timer]
+    - Time transaction took to execute.
+    - Updated for failed and successful transactions.
+    - This does not include commit time, only the time from start until commit is called.
+* tx_with_collision - [Meter]
+    - Rate of transactions with collisions.
+* tx_collisions - [Meter]
+    - Rate of collisions.
+* tx_entries_set - [Meter]
+    - Rate of row/columns set by transaction
+* tx_entries_read - [Meter]
+    - Rate of row/columns read by transaction that existed.
+    - There is currently no count of all reads (including non-existent data)
+* tx_locks_timedout - [Meter]
+    - Rate of timedout locks rolled back by transaction.
+    - These are locks that are held for very long periods by another transaction that appears
to be
+      alive based on zookeeper.
+* tx_locks_dead - [Meter]
+    - Rate of dead locks rolled by a transaction.
+    - These are locks held by a process that appears to be dead according to zookeeper.
+* tx_status_`<STATUS>` - [Meter]
+    - Rate of different ways (i.e `<STATUS>`) a transaction can terminate
+Below are system-wide metrics that are reported for the entire Fluo application. These metrics
+named using the format `fluo.system.APPLICATION.REPORTER_ID.METRIC`.
+* oracle_response_time - [Timer]
+    - Time each RPC call to oracle for stamps took
+* oracle_client_stamps - [Histogram]
+    - Number of stamps requested for each request for stamps to the server
+* oracle_server_stamps - [Histogram]
+    - Number of stamps requested for each request for stamps from a client
+* worker_notifications_queued - [Gauge]
+    - The current number of notifications queued for processing.
+* transactor_committing - [Gauge]
+    - The current number of transactions that are working their way through the commit steps.
+Histograms and Timers have a counter. In the case of a histogram, the counter is the number
of times
+the metric was updated and not a sum of the updates. For example if a request for 5 timestamps
+made to the oracle followed by a request for 3 timestamps, then the count for `oracle_server_stamps`
+would be 2 and the mean would be (5+3)/2.
+## View metrics in Grafana/InfuxDB
+Fluo metrics can be sent to [InfluxDB], a time series database, and made viewable in [Grafana],

+a visualization tool by following the instructions below:
+1.  Follow the standard installation instructions for [InfluxDB] and [Grafana]. As for versions,
+    the instructions below were written using InfluxDB v0.9.4.2 and Grafana v2.5.0.
+2.  Add the following to your InfluxDB configuration to configure it accept metrics in Graphite
+    format from Fluo. The configuration below contains templates that transform the Graphite
+    metrics into a format that is usable in InfluxDB.
+    ```
+    [[graphite]]
+      bind-address = ":2003"
+      enabled = true
+      database = "fluo_metrics"
+      protocol = "tcp"
+      consistency-level = "one"
+      separator = "_"
+      batch-size = 1000
+      batch-pending = 5
+      batch-timeout = "1s"
+      templates = [
+        "fluo.class.*.*.*.*.* ..app.host.measurement.observer.field",
+        "fluo.class.*.*.*.* ..app.host.measurement.observer",
+        "fluo.system.*.*.*.* ..app.host.measurement.field",
+        "fluo.system.*.*.* ..app.host.measurement",
+        "fluo.app.*.*.* ..host.measurement.field",
+        "fluo.app.*.* ..host.measurement",
+      ]
+    ```
+3. Fluo distributes a file called `fluo_metrics_setup.txt` that contains a list of commands
+   setup InfluxDB. These commands will configure an InfluxDB user, retention policies, and
+   continuous queries that downsample data for the historical dashboard in Grafana. Run the
+   below to execute the commands in this file:
+    ```
+    $INFLUXDB_HOME/bin/influx -import -path $FLUO_HOME/contrib/influxdb/fluo_metrics_setup.txt
+    ```
+3. Configure the `fluo-app.properties` of your Fluo application to send Graphite metrics
to InfluxDB.
+   Below is example configuration. Remember to replace `<INFLUXDB_HOST>` with the actual
+    ```
+    fluo.metrics.reporter.graphite.enable=true
+    fluo.metrics.reporter.graphite.host=<INFLUXDB_HOST>
+    fluo.metrics.reporter.graphite.port=2003
+    fluo.metrics.reporter.graphite.frequency=30
+    ```
+    The reporting frequency of 30 sec is required if you are using the provided Grafana dashboards
+    that are configured in the next step.
+4.  Grafana needs to be configured to load dashboard JSON templates from a directory. Fluo
+    distributes two Grafana dashboard templates in its tarball distribution in the directory
+    `contrib/grafana`. Before restarting Grafana, you should copy the templates from your
+    installation to the `dashboards/` directory configured below.
+    ```
+    [dashboards.json]
+    enabled = true
+    path = <GRAFANA_HOME>/dashboards
+    ```
+5.  If you restart Grafana, you will see the Fluo dashboards configured but all of their
charts will
+    be empty unless you have a Fluo application running and configured to send data to InfluxDB.
+    When you start sending data, you may need to refresh the dashboard page in the browser
to start
+    viewing metrics.
+[Grafana]: http://grafana.org/
+[InfluxDB]: https://influxdb.com/
+[fluo-app.properties]: {{ page.github_base }}/modules/distribution/src/main/config/fluo-app.properties
+[Dropwizard]: https://dropwizard.github.io/metrics/3.1.0/
+[MetricsReporter]: {{ page.javadoc_base }}/org/apache/fluo/api/metrics/MetricsReporter.html
+[Observer]: {{ page.javadoc_base }}/org/apache/fluo/api/observer/Observer.html
+[Loader]: {{ page.javadoc_base }}/org/apache/fluo/api/client/Loader.html
+[FluoClient]: {{ page.javadoc_base }}/org/apache/fluo/api/client/FluoClient.html
+[Timer]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#timers
+[Counter]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#counters
+[Histogram]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#histograms
+[Gauge]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#gauges
+[Meter]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#meters
diff --git a/_fluo-docs-1-2/development/applications.md b/_fluo-docs-1-2/development/applications.md
new file mode 100644
index 0000000..a1fb8f9
--- /dev/null
+++ b/_fluo-docs-1-2/development/applications.md
@@ -0,0 +1,314 @@
+title: Applications
+category: development
+order: 1
+Once you have Fluo installed and running on your cluster, you can run Fluo applications consisting
+of [clients and observers][design]. This documentations will shows how to :
+ * Create a Fluo client
+ * Create a Fluo observer
+ * Initialize a Fluo Application
+ * Start and stop a Fluo application (which consists of Oracle and Worker processes)
+## Fluo Maven Dependencies
+For both clients and observers, you will need to include the following in your Maven pom:
+  <groupId>org.apache.fluo</groupId>
+  <artifactId>fluo-api</artifactId>
+  <version>1.2.0</version>
+  <groupId>org.apache.fluo</groupId>
+  <artifactId>fluo-core</artifactId>
+  <version>1.2.0</version>
+  <scope>runtime</scope>
+Fluo provides a classpath command to help users build a runtime classpath. This command along
+the `hadoop jar` command is useful when writing scripts to run Fluo client code. These commands
+allow the scripts to use the versions of Hadoop, Accumulo, and Zookeeper installed on a cluster.
+## Creating a Fluo client
+To create a [FluoClient], you will need to provide it with a [FluoConfiguration] object that
+configured to connect to your Fluo instance.
+If you have access to the [fluo-conn.properties] file that was used to configure your Fluo
instance, you
+can use it to build a [FluoConfiguration] object with all necessary properties:
+FluoConfiguration config = new FluoConfiguration(new File("fluo-conn.properties"));
+You can also create an empty [FluoConfiguration] object and set properties using Java:
+FluoConfiguration config = new FluoConfiguration();
+Once you have [FluoConfiguration] object, pass it to the `newClient()` method of [FluoFactory]
+create a [FluoClient]:
+try(FluoClient client = FluoFactory.newClient(config)){
+  try (Transaction tx = client.newTransaction()) {
+    // read and write some data
+    tx.commit();
+  }
+  try (Snapshot snapshot = client.newSnapshot()) {
+    //read some data
+  }
+It may help to reference the [API javadocs][API] while you are learning the Fluo API.
+## Creating a Fluo observer
+To create an observer, follow these steps:
+1.  Create one or more classes that extend [Observer] like the example below. Please use
[slf4j] for
+    any logging in observers as [slf4j] supports multiple logging implementations. This is
+    necessary as Fluo applications have a hard requirement on [logback] when running in YARN.
+    ```java
+    public class InvertObserver implements Observer {
+      @Override
+      public void process(TransactionBase tx, Bytes row, Column col) throws Exception {
+        // read value
+        Bytes value = tx.get(row, col);
+        // invert row and value
+        tx.set(value, new Column("inv", "data"), row);
+      }
+    }
+    ```
+2.  Create a class that implements [ObserverProvider] like the example below.  The purpose
of this
+    class is associate a set Observers with columns that trigger the observers.  The class
+    register multiple observers.
+    ```java
+    class AppObserverProvider implements ObserverProvider {
+      @Override
+      public void provide(Registry or, Context ctx) {
+        //setup InvertObserver to be triggered when the column obs:data is modified
+        or.forColumn(new Column("obs", "data"), NotificationType.STRONG)
+          .useObserver(new InvertObserver());
+        //Observer is a Functional interface.  So Observers can be written as lambdas.
+        or.forColumn(new Column("new","data"), NotificationType.WEAK)
+          .useObserver((tx,row,col) -> {
+             Bytes combined = combineNewAndOld(tx,row);
+             tx.set(row, new Column("current","data"), combined);
+           });
+      }
+    }
+    ```
+3.  Build a jar containing these classes and include this jar in the `lib/` directory of
your Fluo
+    application.
+4.  Configure your Fluo application to use this observer provider by modifying the Application
section of
+    [fluo-app.properties]. Set `fluo.observer.provider` to the observer provider class name.
+5.  Initialize your Fluo application as described in the next section.  During initialization
+    will obtain the observed columns from the ObserverProvider and persist the columns in
+    These columns persisted in Zookeeper are used by transactions to know when to trigger
+## Initializing a Fluo Application
+Before a Fluo Application can run, it must be initiaized.  Below is an overview of what
+initialization does and some of the properties that may be set for initialization.
+ * **Initialize ZooKeeper** : Each application has its own area in ZooKeeper used for configuration,
+   Oracle state, and worker coordination. All properties, except `fluo.connections.*`, are
+   into ZooKeeper. For example, if `fluo.worker.num.threads=128` was set, then when a worker
+   starts it will read this from ZooKeeper.
+ * **Copy Observer jars to DFS** : Fluo workers processes need the jars containing observers.
+   are provided in one of the following ways.
+   * Set the property `fluo.observer.init.dir` to a local directory containing observer jars.
+     jars in this directory are copied to DFS under `<fluo.dfs.root>/<app name>`.
When a worker is
+     started, the jars are pulled from DFS and added to its classpath.
+   * Set the property `fluo.observer.jars.url` to a directory in DFS containing observer
jars.  No
+     copying is done. When a worker is started, the jars are pulled from this location and
added to
+     its classpath.
+   * Do not set any of the properties above and have the mechanism that starts the worker
+     add the needed jars to the classpath.
+ * **Create Accumulo table** : Each Fluo application creates and configures an Accumulo table.
+   `fluo.accumulo.*` properties determine which Accumulo instance is used. For performance
+   Fluo runs its own code in Accumulo tablet servers. Fluo attempts to copy Fluo jars into
DFS and
+   configure Accumulo to use them. Fluo first checks the property `fluo.accumulo.jars` and
if set,
+   copies the jars listed there. If that property is not set, then Fluo looks on the classpath
+   find jars. Jars are copied to a location under `<fluo.dfs.root>/<app name>`.
+Below are the steps to initialize an application from the command line. It is also possible
+initialize an application using Fluo's Java API.
+1. Create a copy of [fluo-app.properties] for your Fluo application. 
+        cp $FLUO_HOME/conf/fluo-app.properties /path/to/myapp/fluo-app.properties
+2. Edit your copy of [fluo-app.properties] and make sure to set the following:
+    * Class name of your ObserverProvider
+    * Paths to your Fluo observer jars
+    * Accumulo configuration
+    * DFS configuration
+   When configuring the observer section of fluo-app.properties, you can configure your instance
for the
+   [phrasecount] application if you have not created your own application. See the [phrasecount]
+   example for instructions. You can also choose not to configure any observers but your
workers will
+   be idle when started.
+3. Run the command below to initialize your Fluo application. Change `myapp` to your application
+        fluo init myapp /path/to/myapp/fluo-app.properties
+   A Fluo application only needs to be initialized once. After initialization, the Fluo application
+   name is used to start/stop the application and scan the Fluo table.
+4. Run `fluo list` which connects to Fluo and lists applications to verify initialization.
+5. Run `fluo config myapp` to see what configuration is stored in ZooKeeper.
+## Starting your Fluo application
+Follow the instructions below to start a Fluo application which contains an oracle and multiple
+1. Configure [fluo-env.sh] and [fluo-conn.properties] if you have not already.
+2. Run Fluo application processes using the `fluo oracle` and `fluo worker` commands. Fluo
+   are typically run with one oracle process and multiple worker processes. The commands
below will start
+   a Fluo oracle and two workers on your local machine:
+        fluo oracle myapp &> oracle.log &
+        fluo worker myapp &> worker1.log &
+        fluo worker myapp &> worker2.log &
+   The commands will retrieve your application configuration and observer jars (using your
+   application name) before starting the oracle or worker process.
+If you want to distribute the processes of your Fluo application across a cluster, you will
need install
+Fluo on every node where you want to run a Fluo process and follow the instructions above
on each node.
+## Managing your Fluo application
+When you have data in your Fluo application, you can view it using the command `fluo scan
+Pipe the output to `less` using the command `fluo scan myapp | less` if you want to page
through the data.
+To list all Fluo applications, run `fluo list`.
+To stop your Fluo application, run `jps -m | grep Fluo` to find process IDs and use `kill`
to stop them.
+## Running application code
+The `fluo exec <app name> <class> {arguments}` provides an easy way to execute
application code. It
+will execute a class with a main method if a jar containing the class is included with the
+jars configured at initialization. When the class is run, Fluo classes and dependencies will
be on 
+the classpath. The `fluo exec` command can inject the applications configuration if the class
+written in the following way. Defining the injection point is optional.
+import javax.inject.Inject;
+public class AppCommand {
+  //when run with fluo exec command, the applications configuration will be injected
+  @Inject
+  private static FluoConfiguration fluoConfig;
+  public static void main(String[] args) throws Exception {
+    try(FluoClient fluoClient = FluoFactory.newClient(fluoConfig)) {
+      //do stuff with Fluo
+    }
+  }
+## Application Configuration
+For configuring observers, fluo provides a simple mechanism to set and access application
+configuration.  See the javadoc on [FluoClient].getAppConfiguration() for more details.
+## Debugging Applications
+While monitoring [Fluo metrics][metrics] can detect problems (like too many transaction collisions)
+in a Fluo application, [metrics][metrics] may not provide enough information to debug the
root cause
+of the problem. To help debug Fluo applications, low-level logging of transactions can be
turned on
+by setting the following loggers to TRACE:
+| Logger               | Level | Information                                            
+| fluo.tx            | TRACE | Provides detailed information about what transactions read
and wrote                               |
+| fluo.tx.summary    | TRACE | Provides a one line summary about each transaction executed
+| fluo.tx.collisions | TRACE | Provides details about what data was involved When a transaction
collides with another transaction |
+| fluo.tx.scan       | TRACE | Provides logging for each cell read by a scan.  Scan summary
logged at `fluo.tx` level.  This allows suppression of `fluo.tx.scan` while still seeing summary.
+Below is an example log after setting `fluo.tx` to TRACE. The number following `txid: ` is
+transactions start timestamp from the Oracle.
+2015-02-11 18:24:05,341 [fluo.tx ] TRACE: txid: 3 begin() thread: 198
+2015-02-11 18:24:05,343 [fluo.tx ] TRACE: txid: 3 class: com.SimpleLoader
+2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 3 get(4333, stat count ) -> null
+2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 3 set(4333, stat count , 1)
+2015-02-11 18:24:05,441 [fluo.tx ] TRACE: txid: 3 commit() -> SUCCESSFUL commitTs: 4
+2015-02-11 18:24:05,341 [fluo.tx ] TRACE: txid: 5 begin() thread: 198
+2015-02-11 18:24:05,442 [fluo.tx ] TRACE: txid: 3 close()
+2015-02-11 18:24:05,343 [fluo.tx ] TRACE: txid: 5 class: com.SimpleLoader
+2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 5 get(4333, stat count ) -> 1
+2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 5 set(4333, stat count , 2)
+2015-02-11 18:24:05,441 [fluo.tx ] TRACE: txid: 5 commit() -> SUCCESSFUL commitTs: 6
+2015-02-11 18:24:05,442 [fluo.tx ] TRACE: txid: 5 close()
+The log above traces the following sequence of events.
+* Transaction T1 has a start timestamp of `3`
+* Thread with id `198` is executing T1, its running code from the class `com.SimpleLoader`
+* T1 reads row `4333` and column `stat count` which does not exist
+* T1 sets row `4333` and column `stat count` to `1`
+* T1 commits successfully and its commit timestamp from the Oracle is `4`.
+* Transaction T2 has a start timestamp of `5` (because its `5` > `4` it can see what T1
+* T2 reads a value of `1` for row `4333` and column `stat count`
+* T2 sets row `4333` and column `stat count` to `2`
+* T2 commits successfully with a commit timestamp of `6`
+Below is an example log after only setting `fluo.tx.collisions` to TRACE. This setting will
only log
+trace information when a collision occurs. Unlike the previous example, what the transaction
+and wrote is not logged. This shows that a transaction with a start timestamp of `106` and
a class
+name of `com.SimpleLoader` collided with another transaction on row `r1` and column `fam1
+2015-02-11 18:17:02,639 [tx.collisions] TRACE: txid: 106 class: com.SimpleLoader
+2015-02-11 18:17:02,639 [tx.collisions] TRACE: txid: 106 collisions: {r1=[fam1 qual1 ]}
+When applications read and write arbitrary binary data, this does not log so well. In order
to make
+the trace logs human readable, non ASCII chars are escaped using hex. The convention used
it `\xDD`
+where D is a hex digit. Also the `\` character is escaped to make the output unambiguous.
+[design]: {{ page.docs_base }}/getting-started/design/
+[FluoFactory]: {{ page.javadoc_base}}/org/apache/fluo/api/client/FluoFactory.html
+[FluoClient]: {{ page.javadoc_base}}/org/apache/fluo/api/client/FluoClient.html
+[FluoConfiguration]: {{ page.javadoc_base}}/org/apache/fluo/api/config/FluoConfiguration.html
+[Observer]: {{ page.javadoc_base}}/org/apache/fluo/api/observer/Observer.html
+[ObserverProvider]: {{ page.javadoc_base}}/org/apache/fluo/api/observer/ObserverProvider.html
+[fluo-conn.properties]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-conn.properties
+[fluo-app.properties]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-app.properties
+[API]: https://fluo.apache.org/apidocs/
+[metrics]: {{ page.docs_base }}/administration/metrics/
+[slf4j]: http://www.slf4j.org/
+[logback]: http://logback.qos.ch/
+[phrasecount]: https://github.com/astralway/phrasecount
+[fluo-env.sh]: {{ page.github_base}}/modules/distribution/src/main/config/fluo-env.sh
diff --git a/_fluo-docs-1-2/getting-started/design.md b/_fluo-docs-1-2/getting-started/design.md
new file mode 100644
index 0000000..00d6f7a
--- /dev/null
+++ b/_fluo-docs-1-2/getting-started/design.md
@@ -0,0 +1,45 @@
+title: Design
+category: getting-started
+order: 1
+The diagram below provides an overview of Apache Fluo's design.
+## Fluo Application
+A **Fluo application** maintains a large scale computation using a series of small transactional
+updates. Fluo applications store their data in a **Fluo table** which has a similar structure
+column, value) to an **Accumulo table** except that a Fluo table has no timestamps. A Fluo
+is implemented using an Accumulo table. While you could scan the Accumulo table used to implement
+a Fluo table using an Accumulo client, you would read extra implementation-related data in
+to your data. Therefore, developers should only interact with the data in a Fluo table by
+Fluo client or observer code:
+* **Clients** ingest data or interact with Fluo from external applications (REST services,
+  crawlers, etc).  These are generally user started process that use the Fluo API.
+* **Observers** are user provided functions run by Fluo Workers that execute transactions
in response to notifications. Notifications are set by Fluo transactions, executing in a client
or observer, when a requested column is modified. 
+Multiple Fluo applications can run on a cluster at the same time. Fluo applications
+consist of an oracle process and a configurable number of worker processes:
+* The **Oracle** process allocates timestamps for transactions. While only one Oracle is
+  Fluo can be configured to run extra Oracles that can take over if the primary Oracle fails.
+* **Worker** processes run user code (called **observers**) that perform transactions. All
+   run the same observers. The number of worker instances are configured to handle the processing
+   workload.
+## Fluo Dependencies
+Fluo requires the following software to be running on the cluster:
+* **Accumulo** - Fluo stores its data in Accumulo and uses Accumulo's conditional mutations
+  transactions.
+* **Hadoop** - Each Fluo application run its oracle and worker processes as Hadoop YARN
+  applications. HDFS is also required for Accumulo.
+* **Zookeeper** - Fluo stores its metadata and state information in Zookeeper. Zookeeper
is also
+  required for Accumulo.
+[1]: /resources/docs/fluo-architecture.png
diff --git a/_fluo-docs-1-2/getting-started/install.md b/_fluo-docs-1-2/getting-started/install.md
new file mode 100644
index 0000000..68de78b
--- /dev/null
+++ b/_fluo-docs-1-2/getting-started/install.md
@@ -0,0 +1,115 @@
+title: Installation
+category: getting-started
+order: 2
+Instructions for installing Apache Fluo and starting a Fluo application on a cluster where
+Accumulo, Hadoop & Zookeeper are running.  If you need help setting up these dependencies,
see the
+[related projects page][related] for external projects that may help.
+## Requirements
+Before you install Fluo, the following software must be installed and running on your local
+or cluster:
+| Software    | Recommended Version | Minimum Version |
+| [Accumulo]  | 1.7.2               | 1.6.1           |
+| [Hadoop]    | 2.7.2               | 2.6.0           |
+| [Zookeeper] | 3.4.8               |                 |
+| [Java]      | JDK 8               | JDK 8           |
+## Obtain a distribution
+Before you can install Fluo, you will need to obtain a distribution tarball. It is recommended
+you download the [latest release][release]. You can also build a distribution from the master
+branch by following these steps which create a tarball in `modules/distribution/target`:
+    git clone https://github.com/apache/fluo.git
+    cd fluo/
+    mvn package
+## Install Fluo
+After you obtain a Fluo distribution tarball, follow these steps to install Fluo.
+1.  Choose a directory with plenty of space and untar the distribution:
+        tar -xvzf fluo-1.2.0-bin.tar.gz
+        cd fluo-1.2.0
+    The distribution contains a `fluo` script in `bin/` that administers Fluo and the
+    following configuration files in `conf/`:
+    | Configuration file           | Description                                        
+    |------------------------------|----------------------------------------------------------------------------------------------|
+    | [fluo-env.sh]                | Configures classpath for `fluo` script. Required for
all commands.                           |
+    | [fluo-conn.properties]       | Configures connection to Fluo. Required for all commands.
+    | [fluo-app.properties]        | Template for configuration file passed to `fluo init`
when initializing Fluo application.    |
+    | [log4j.properties]           | Configures logging                                 
+    | [fluo.properties.deprecated] | Deprecated Fluo configuration file. Replaced by fluo-conn.properties
and fluo-app.properties |
+2.  Configure [fluo-env.sh] to set up your classpath using jars from the versions of Hadoop,
Accumulo, and
+Zookeeper that you are using. Choose one of the two ways below to make these jars available
to Fluo:
+    * Set `HADOOP_PREFIX`, `ACCUMULO_HOME`, and `ZOOKEEPER_HOME` in your environment or configure
+    these variables in [fluo-env.sh]. Fluo will look in these locations for jars.
+    * Run `./lib/fetch.sh ahz` to download Hadoop, Accumulo, and Zookeeper jars to `lib/ahz`
+    configure [fluo-env.sh] to look in this directory. By default, this command will download
+    default versions set in [lib/ahz/pom.xml]. If you are not using the default versions,
you can
+    override them:
+            ./lib/fetch.sh ahz -Daccumulo.version=1.7.2 -Dhadoop.version=2.7.2 -Dzookeeper.version=3.4.8
+3. Fluo needs more dependencies than what is available from Hadoop, Accumulo, and Zookeeper.
+   extra dependencies need to be downloaded to `lib/` using the command below:
+        ./lib/fetch.sh extra
+You are now ready to use the `fluo` script.
+## Fluo command script
+The Fluo command script is located at `bin/fluo` of your Fluo installation. All Fluo commands
+invoked by this script.
+Modify and add the following to your `~/.bashrc` if you want to be able to execute the fluo
+from any directory:
+    export PATH=/path/to/fluo-1.2.0/bin:$PATH
+Source your `.bashrc` for the changes to take effect and test the script
+    source ~/.bashrc
+    fluo
+Running the script without any arguments prints a description of all commands.
+    ./bin/fluo
+## Tuning Accumulo
+Fluo will reread the same data frequently when it checks conditions on mutations. When Fluo
+initializes a table it enables data caching to make this more efficient. However you may
need to
+increase the amount of memory available for caching in the tserver by increasing
+`tserver.cache.data.size`. Increasing this may require increasing the maximum tserver java
heap size
+in `accumulo-env.sh`.
+Fluo will run many client threads, will want to ensure the tablet server has enough threads.
+probably increase the `tserver.server.threads.minimum` Accumulo setting.
+Using at least Accumulo 1.6.1 is recommended because multiple performance bugs were fixed.
+[Accumulo]: https://accumulo.apache.org/
+[Hadoop]: http://hadoop.apache.org/
+[Zookeeper]: http://zookeeper.apache.org/
+[Java]: http://openjdk.java.net/
+[related]: https://fluo.apache.org/related-projects/
+[release]: https://fluo.apache.org/download/
+[fluo-conn.properties]: {{ page.github_base }}/modules/distribution/src/main/config/fluo-conn.properties
+[fluo-app.properties]: {{ page.github_base }}/modules/distribution/src/main/config/fluo-app.properties
+[log4j.properties]: {{ page.github_base }}/modules/distribution/src/main/config/log4j.properties
+[fluo.properties.deprecated]: {{ page.github_base }}/modules/distribution/src/main/config/fluo.properties.deprecated
+[fluo-env.sh]: {{ page.github_base }}/modules/distribution/src/main/config/fluo-env.sh
+[lib/ahz/pom.xml]: {{ page.github_base }}/modules/distribution/src/main/lib/ahz/pom.xml
diff --git a/_fluo-docs-1-2/index.md b/_fluo-docs-1-2/index.md
new file mode 100644
index 0000000..349b42d
--- /dev/null
+++ b/_fluo-docs-1-2/index.md
@@ -0,0 +1,4 @@
+title: Apache Fluo documentation
+redirect_to: getting-started/design
diff --git a/_layouts/default.html b/_layouts/default.html
index 15f5342..dcd034c 100644
--- a/_layouts/default.html
+++ b/_layouts/default.html
@@ -34,7 +34,13 @@
           <ul class="navbar-nav nav">
             <li><a href="{{ site.baseurl }}/release/">Releases</a></li>
             <li><a href="{{ site.baseurl }}/tour/">Tour</a></li>
-            <li><a href="{{ site.baseurl }}/docs/">Docs</a></li>
+            <li class="dropdown">
+              <a class="dropdown-toggle" data-toggle="dropdown" href="#">Docs<span
+              <ul class="dropdown-menu">
+                <li><a href="{{ site.baseurl }}/docs/fluo/{{ site.latest_fluo_release
+                <li><a href="{{ site.baseurl }}/docs/fluo-recipes/{{ site.latest_recipes_release
}}/">Fluo Recipes</a></li>
+              </ul>
+            </li>
             <li><a href="{{ site.baseurl }}/api/">API</a></li>
             <li class="dropdown">
               <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span
diff --git a/_layouts/fluo-docs-1.2.html b/_layouts/fluo-docs-1.2.html
new file mode 100644
index 0000000..c53ffd3
--- /dev/null
+++ b/_layouts/fluo-docs-1.2.html
@@ -0,0 +1,55 @@
+layout: default
+<div class="row">
+  <div class="col-md-2">
+    <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true"
+      <div class="panel panel-default">
+      {% assign mydocs = site.fluo-docs-1-2 | group_by: 'category' %}
+      {% assign categories = "getting-started,development,administration" | split: "," %}
+      {% for pcat in categories %}
+        {% for dcat in mydocs %}
+          {% if pcat == dcat.name %}
+            <div class="panel-heading" role="tab" id="headingOne">
+              <h4 class="panel-title">
+                <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse{{
pcat }}" aria-expanded="{% if pcat == page.category %}true{% else %}false{% endif %}" aria-controls="collapse{{
pcat }}">
+                  {{ pcat | capitalize | replace: "-", " " }}
+                </a>
+              </h4>
+            </div>
+            <div id="collapse{{pcat}}" class="panel-collapse collapse{% if pcat == page.category
%} in{% endif %}" role="tabpanel" aria-labelledby="headingOne">
+              <div class="panel-body">
+                {% assign items = dcat.items | sort: 'order' %}
+                {% for item in items %}
+                <div class="row doc-sidebar-link"><a href="{{ item.url }}">{{
item.title }}</a></div>
+                {% endfor %}
+              </div>
+            </div>
+          {% endif %}
+        {% endfor %}
+      {% endfor %}
+      </div>
+    </div>
+  </div>
+  <div class="col-md-10">
+    {% if page.category %}
+    <p>Fluo {{ page.version }} documentation &nbsp;&gt;&gt;&nbsp; {{
page.category | capitalize | replace: "-", " " }} &nbsp;&gt;&gt;&nbsp; {{
page.title }}</p>
+    {% endif %}
+    <div class="alert alert-danger" style="margin-bottom: 0px;" role="alert">This documentation
is for a future release of Fluo! <a href="{{ site.baseurl }}/docs/fluo/{{ site.latest_fluo_release
}}/">View documentation for the latest release</a>.</div>
+    {% unless page.nodoctitle %}
+    <div class="row">
+      <div class="col-md-10"><h1>{{ page.title }}</h1></div>
+      <div class="col-md-2"><a class="pull-right" style="margin-top: 25px;" href="https://github.com/apache/fluo-website/edit/master/{{
page.path }}" role="button"><i class="glyphicon glyphicon-pencil"></i> <small>Edit
this page</small></a></div>
+    </div>  
+    {% endunless %}
+    {{ content }}
+    <div class="row" style="margin-top: 20px;">
+      <div class="col-md-10"><strong>Find documentation for all Fluo releases
in the <a href="{{ site.baseurl }}/docs/">archive</strong></div>
+      <div class="col-md-2"><a class="pull-right" href="https://github.com/apache/fluo-website/edit/master/{{
page.path }}" role="button"><i class="glyphicon glyphicon-pencil"></i> <small>Edit
this page</small></a></div>
+    </div>  
+  </div>
diff --git a/css/fluo.scss b/css/fluo.scss
index 150608e..6e9d7fe 100644
--- a/css/fluo.scss
+++ b/css/fluo.scss
@@ -30,6 +30,12 @@ code {
   margin-left: 0px;
+.doc-sidebar-link {
+  font-size: 14px;
+  margin-bottom: 10px;
+  margin-left: 0px;
 tr:nth-child(even) {background: #F3F3F3}
 tr:nth-child(odd) {background: #FFF}
diff --git a/docs/fluo/1.0.0-incubating/index.md b/docs/fluo/1.0.0-incubating/index.md
index 4373f85..9ecbe94 100644
--- a/docs/fluo/1.0.0-incubating/index.md
+++ b/docs/fluo/1.0.0-incubating/index.md
@@ -33,6 +33,8 @@ Below are helpful resources for Fluo application developers:
 *  [Metrics] - Fluo metrics are visible via JMX by default but can be configured to send
to Graphite
    or Ganglia
+**Find documentation for all Fluo releases in the [archive](/docs/)**.
 [related]: /related-projects/
 [tour]: /tour/
 [accumulo]: https://accumulo.apache.org
diff --git a/docs/fluo/1.1.0-incubating/index.md b/docs/fluo/1.1.0-incubating/index.md
index 9f5e520..18914d7 100644
--- a/docs/fluo/1.1.0-incubating/index.md
+++ b/docs/fluo/1.1.0-incubating/index.md
@@ -33,6 +33,8 @@ Below are helpful resources for Fluo application developers:
 *  [Metrics] - Fluo metrics are visible via JMX by default but can be configured to send
to Graphite
    or Ganglia
+**Find documentation for all Fluo releases in the [archive](/docs/)**.
 [fluo]: https://fluo.apache.org/
 [related]: https://fluo.apache.org/related-projects/
 [tour]: https://fluo.apache.org/tour/
diff --git a/docs/index.md b/docs/index.md
index b7ace5f..9e718e5 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,13 +1,11 @@
 layout: page
-title: Documentation
+title: Documentation Archive
   - /docs/fluo/
   - /docs/fluo-recipes/
-For a general overview of Fluo, take the [Fluo tour](/tour/).
 [Apache Fluo] and [Apache Fluo Recipes] have separate documentation as they are different
repositories with their own release cycle.
 ### Apache Fluo documentation
diff --git a/resources/docs/fluo-architecture.odg b/resources/docs/fluo-architecture.odg
new file mode 100644
index 0000000..fb2a9ad
Binary files /dev/null and b/resources/docs/fluo-architecture.odg differ
diff --git a/resources/docs/fluo-architecture.png b/resources/docs/fluo-architecture.png
new file mode 100644
index 0000000..3ba96fd
Binary files /dev/null and b/resources/docs/fluo-architecture.png differ

To stop receiving notification emails like this one, please contact
['"commits@fluo.apache.org" <commits@fluo.apache.org>'].

View raw message