fluo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mwa...@apache.org
Subject [1/2] incubator-fluo git commit: Improved documentation for viewing Fluo logs
Date Wed, 09 Nov 2016 21:16:13 GMT
Repository: incubator-fluo
Updated Branches:
  refs/heads/master 8e08d89a5 -> b56868134

Improved documentation for viewing Fluo logs

* Improved docs on creating and logging in Observers

Project: http://git-wip-us.apache.org/repos/asf/incubator-fluo/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-fluo/commit/4f19e887
Tree: http://git-wip-us.apache.org/repos/asf/incubator-fluo/tree/4f19e887
Diff: http://git-wip-us.apache.org/repos/asf/incubator-fluo/diff/4f19e887

Branch: refs/heads/master
Commit: 4f19e887da29358032bc8ec339ecf8872ba11d25
Parents: 1d243c3
Author: Mike Walch <mwalch@apache.org>
Authored: Tue Nov 1 15:10:03 2016 -0400
Committer: Mike Walch <mwalch@apache.org>
Committed: Wed Nov 9 15:59:06 2016 -0500

 docs/applications.md | 24 +++++++++++++++++++++++-
 docs/install.md      | 35 ++++++++++++++++++++++++++++++++---
 2 files changed, 55 insertions(+), 4 deletions(-)

diff --git a/docs/applications.md b/docs/applications.md
index b7dd02b..7471a3d 100644
--- a/docs/applications.md
+++ b/docs/applications.md
@@ -100,7 +100,27 @@ public class AppCommand {
 To create an observer, follow these steps:
-1.  Create a class that extends [AbstractObserver].
+1. Create a class that extends [AbstractObserver] like the example below. Please use [slf4j]
+   any logging in observers as [slf4j] supports multiple logging implementations. This is
+   necessary as Fluo applications have a hard requirement on [logback] when running in YARN.
+    ```java
+    public class InvertObserver extends AbstractObserver {
+      @Override
+      public void process(TransactionBase tx, Bytes row, Column col) throws Exception {
+        // read value
+        Bytes value = tx.get(row, col);
+        // invert row and value
+        tx.set(value, new Column("inv", "data"), row);
+      }
+      @Override
+      public ObservedColumn getObservedColumn() {
+        return new ObservedColumn(new Column("obs", "data"), NotificationType.STRONG);
+      }
+    }
+    ```
 2.  Build a jar containing this class and include this jar in the `lib/` directory of your
 3.  Configure your Fluo instance to use this observer by modifying the Observer section of
@@ -178,3 +198,5 @@ where D is a hex digit. Also the `\` character is escaped to make the
output una
 [fluo.properties]: ../modules/distribution/src/main/config/fluo.properties
 [API]: https://fluo.apache.org/apidocs/
 [metrics]: metrics.md
+[slf4j]: http://www.slf4j.org/
+[logback]: http://logback.qos.ch/

diff --git a/docs/install.md b/docs/install.md
index 18f4380..818249f 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -157,10 +157,37 @@ For more detailed information on the YARN containers running Fluo:
     fluo info myapp
-You can also use `yarn application -list` to check the status of your Fluo instance in YARN.
-are viewable within YARN.
+You can also use `yarn application -list` to check the status of your Fluo instance in YARN.

-When you have data in your fluo instance, you can view it using the command `fluo scan`.
Pipe the
+## View Fluo application logs
+Fluo application logs are viewable within YARN using the methods below:
+1. View logs using the web interface of the the YARN resource manager
+(`http://<resource manager>:8088/cluster`). First, click on the application ID (i.e
+of your Fluo application and then click on the latest attempt ID (i.e `appattempt_*`). You
+see a list of containers. There should be a container for the application master (typically
+container 1), a Fluo oracle (typically container 2), and Fluo workers (containers 3+). You
can view
+the log files produced by a container by clicking on its 'logs' link. Logs from Fluo observers
+be in the `worker_*.log` file for each of your worker containers. 
+2. View logs on disk in the directory specified by the YARN property `yarn.nodemanager.log-dirs`
+your YARN configuration `yarn-site.xml` (see [yarn-default.xml] for more info about this
+If you are running Fluo on single machine, this method works well. If you are running Fluo
on a
+cluster, your containers and their logs will be spread across the cluster.
+When running Fluo in YARN, Fluo must use [logback] for logging (due to a hard requirement
by Twill).
+Logback is configured using `/path/to/fluo/apps/<app_name>/conf/logback.xml`. You should
review this
+configuration but the root logger is configured by default to print any message that is debug
+or higher.
+In addition the `*.log` files, Fluo oracle and worker containers will have `stdout` and `stderr`
+files. These files may have helpful error messages. Especially, if a process failed to start
+or calls were made to `System.out` or `System.err` in your application.
+## View Fluo application data
+When you have data in your Fluo application, you can view it using the command `fluo scan`.
Pipe the
 output to `less` using the command `fluo scan | less` if you want to page through the data.
 ## Stop your Fluo application
@@ -221,3 +248,5 @@ node in your cluster.
 [fluo.properties]: ../modules/distribution/src/main/config/fluo.properties
 [fluo-env.sh]: ../modules/distribution/src/main/config/fluo-env.sh
 [lib/ahz/pom.xml]: ../modules/distribution/src/main/lib/ahz/pom.xml
+[yarn-default.xml]: https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
+[logback]: http://logback.qos.ch/

View raw message