accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From e..@apache.org
Subject git commit: ACCUMULO-1502 adding Jared Winick's doc and examples
Date Thu, 12 Sep 2013 18:33:32 GMT
Updated Branches:
  refs/heads/1.5.1-SNAPSHOT 90eb75129 -> 881c182a1


ACCUMULO-1502 adding Jared Winick's doc and examples


Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/881c182a
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/881c182a
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/881c182a

Branch: refs/heads/1.5.1-SNAPSHOT
Commit: 881c182a117838a30b21563813b07028eeba86f0
Parents: 90eb751
Author: Eric Newton <eric.newton@gmail.com>
Authored: Thu Sep 12 13:28:19 2013 -0400
Committer: Eric Newton <eric.newton@gmail.com>
Committed: Thu Sep 12 13:28:19 2013 -0400

----------------------------------------------------------------------
 .../chapters/administration.tex                 | 121 +++++++++++++++
 examples/simple/pom.xml                         |   4 +
 .../simple/client/TraceDumpExample.java         |  77 ++++++++++
 .../examples/simple/client/TracingExample.java  | 154 +++++++++++++++++++
 4 files changed, 356 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/accumulo/blob/881c182a/docs/src/main/latex/accumulo_user_manual/chapters/administration.tex
----------------------------------------------------------------------
diff --git a/docs/src/main/latex/accumulo_user_manual/chapters/administration.tex b/docs/src/main/latex/accumulo_user_manual/chapters/administration.tex
index d0c5ed4..147e522 100644
--- a/docs/src/main/latex/accumulo_user_manual/chapters/administration.tex
+++ b/docs/src/main/latex/accumulo_user_manual/chapters/administration.tex
@@ -192,6 +192,127 @@ The Accumulo Master provides an interface for monitoring the status
and health o
 Accumulo components. This interface can be accessed by pointing a web browser to\\
 \texttt{http://accumulomaster:50095/status}
 
+\section{Tracing}
+It can be difficult to determine why some operations are taking longer
+than expected. For example, you may be looking up items with very low
+latency, but sometimes the lookups take much longer. Determining the
+cause of the delay is difficult because the system is distributed, and
+the typical lookup is fast.
+
+Accumulo has been instrumented to record the time that various
+operations take when tracing is turned on. The fact that tracing is
+enabled follows all the requests made on behalf of the user throughout
+the distributed infrastructure of accumulo, and across all threads of
+execution.
+
+These time spans will be inserted into the \texttt{trace} table in
+Accumulo. You can browse recent traces from the Accumulo monitor
+page. You can also read the \texttt{trace} table directly like any
+other table.
+
+The design of Accumulo's distributed tracing follows that of
+\href{http://research.google.com/pubs/pub36356.html}{Google's Dapper}.
+
+\subsection{Tracers}
+To collect traces, Accumulo needs at least one server listed in
+\\\texttt{\$ACCUMULO\_HOME/conf/tracers}. The server collects traces
+from clients and writes them to the \texttt{trace} table. The Accumulo
+user that the tracer connects to Accumulo with can be configured with
+the following properties
+
+\begin{verbatim}
+trace.user
+trace.token.property.password
+\end{verbatim}
+
+\subsection{Instrumenting a Client}
+Tracing can be used to measure a client operation, such as a scan, as
+the operation traverses the distributed system. To enable tracing for
+your application call
+
+\begin{verbatim}
+DistributedTrace.enable(instance, new ZooReader(instance), hostname, "myApplication");
+\end{verbatim}
+
+Once tracing has been enabled, a client can wrap an operation in a trace.
+
+\begin{verbatim}
+Trace.on("Client Scan");
+BatchScanner scanner = conn.createBatchScanner(...);
+// Configure your scanner
+for (Entry entry : scanner) {
+}
+Trace.off();
+\end{verbatim}
+
+Additionally, the user can create additional Spans within a Trace.
+\begin{verbatim}
+Trace.on("Client Update");
+...
+Span readSpan = Trace.start("Read");
+...
+readSpan.stop();
+...
+Span writeSpan = Trace.start("Write");
+...
+writeSpan.stop();
+Trace.off();
+\end{verbatim}
+
+Like Dapper, Accumulo tracing supports user defined annotations to associate additional data
with a Trace.
+\begin{verbatim}
+...
+int numberOfEntriesRead = 0;
+Span readSpan = Trace.start("Read");
+// Do the read, update the counter
+...
+readSpan.data("Number of Entries Read", String.valueOf(numberOfEntriesRead));
+\end{verbatim}
+
+Some client operations may have a high volume within your
+application. As such, you may wish to only sample a percentage of
+operations for tracing. As seen below, the CountSampler can be used to
+help enable tracing for 1-in-1000 operations
+\begin{verbatim}
+Sampler sampler = new CountSampler(1000);
+...
+if (sampler.next()) {
+  Trace.on("Read");
+}
+...
+Trace.offNoFlush();
+\end{verbatim}
+
+It should be noted that it is safe to turn off tracing even if it
+isn't currently active. The Trace.offNoFlush() should be used if the
+user does not wish to have Trace.off() block while flushing trace
+data.
+
+\subsection{Viewing Collected Traces}
+To view collected traces, use the "Recent Traces" link on the Monitor
+UI. You can also programmatically access and print traces using the
+\texttt{TraceDump} class.
+
+\subsection{Tracing from the Shell}
+You can enable tracing for operations run from the shell by using the
+\texttt{trace on} and \texttt{trace off} commands.
+
+\begin{verbatim}
+root@test test> trace on
+root@test test> scan
+a b:c []    d
+root@test test> trace off
+Waiting for trace information
+Waiting for trace information
+Trace started at 2013/08/26 13:24:08.332
+Time  Start  Service@Location       Name
+ 3628+0      shell@localhost shell:root
+    8+1690     shell@localhost scan
+    7+1691       shell@localhost scan:location
+    6+1692         tserver@localhost startScan
+    5+1692           tserver@localhost tablet read ahead 6
+\end{verbatim}
+
 \section{Logging}
 Accumulo processes each write to a set of log files. By default these are found under\\
 \texttt{\$ACCUMULO/logs/}.

http://git-wip-us.apache.org/repos/asf/accumulo/blob/881c182a/examples/simple/pom.xml
----------------------------------------------------------------------
diff --git a/examples/simple/pom.xml b/examples/simple/pom.xml
index c721f38..ff9376c 100644
--- a/examples/simple/pom.xml
+++ b/examples/simple/pom.xml
@@ -34,6 +34,10 @@
       <artifactId>accumulo-core</artifactId>
     </dependency>
     <dependency>
+      <groupId>org.apache.zookeeper</groupId>
+      <artifactId>zookeeper</artifactId>
+    </dependency>
+    <dependency>
       <groupId>commons-cli</groupId>
       <artifactId>commons-cli</artifactId>
       <scope>provided</scope>

http://git-wip-us.apache.org/repos/asf/accumulo/blob/881c182a/examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TraceDumpExample.java
----------------------------------------------------------------------
diff --git a/examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TraceDumpExample.java
b/examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TraceDumpExample.java
new file mode 100644
index 0000000..2947e0e
--- /dev/null
+++ b/examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TraceDumpExample.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.examples.simple.client;
+
+import org.apache.accumulo.core.cli.ClientOnDefaultTable;
+import org.apache.accumulo.core.cli.ScannerOpts;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.trace.TraceDump;
+import org.apache.accumulo.core.trace.TraceDump.Printer;
+import org.apache.hadoop.io.Text;
+
+import com.beust.jcommander.Parameter;
+
+/**
+ * Example of using the TraceDump class to print a formatted view of a Trace
+ *
+ */
+public class TraceDumpExample {
+	
+	static class Opts extends ClientOnDefaultTable {
+		public Opts() {
+			super("trace");
+		}
+
+		@Parameter(names = {"--traceid"}, description = "The hex string id of a given trace, for
example 16cfbbd7beec4ae3")
+		public String traceId = "";
+	}
+	
+	public void dump(Opts opts) throws TableNotFoundException, AccumuloException, AccumuloSecurityException
{
+	
+		if (opts.traceId.isEmpty()) {
+			throw new IllegalArgumentException("--traceid option is required");
+		}
+		
+		Scanner scanner = opts.getConnector().createScanner(opts.getTableName(), opts.auths);
+		scanner.setRange(new Range(new Text(opts.traceId)));
+		TraceDump.printTrace(scanner, new Printer() {
+			public void print(String line) {
+				System.out.println(line);
+			}
+		});
+	}
+	
+	/**
+	 * @param args
+	 * @throws AccumuloSecurityException 
+	 * @throws AccumuloException 
+	 * @throws TableNotFoundException 
+	 */
+	public static void main(String[] args) throws TableNotFoundException, AccumuloException,
AccumuloSecurityException {
+		TraceDumpExample traceDumpExample = new TraceDumpExample();
+		Opts opts = new Opts();
+		ScannerOpts scannerOpts = new ScannerOpts();
+		opts.parseArgs(TraceDumpExample.class.getName(), args, scannerOpts);
+
+		traceDumpExample.dump(opts);
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/accumulo/blob/881c182a/examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TracingExample.java
----------------------------------------------------------------------
diff --git a/examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TracingExample.java
b/examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TracingExample.java
new file mode 100644
index 0000000..46752c8
--- /dev/null
+++ b/examples/simple/src/main/java/org/apache/accumulo/examples/simple/client/TracingExample.java
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.accumulo.examples.simple.client;
+
+import java.io.IOException;
+import java.util.Map.Entry;
+
+import org.apache.accumulo.core.cli.ClientOnDefaultTable;
+import org.apache.accumulo.core.cli.ScannerOpts;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.BatchWriterConfig;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableExistsException;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.core.trace.DistributedTrace;
+import org.apache.accumulo.fate.zookeeper.ZooReader;
+import org.apache.accumulo.trace.instrument.Span;
+import org.apache.accumulo.trace.instrument.Trace;
+
+import com.beust.jcommander.Parameter;
+
+/**
+ * A simple example showing how to use the distributed tracing API in client code
+ *
+ */
+public class TracingExample {
+	
+	private static final String DEFAULT_TABLE_NAME = "test";
+	
+	static class Opts extends ClientOnDefaultTable {
+	    @Parameter(names = {"-C", "--createtable"}, description = "create table before doing
anything")
+	    boolean createtable = false;
+	    @Parameter(names = {"-D", "--deletetable"}, description = "delete table when finished")
+	    boolean deletetable = false;
+	    @Parameter(names = {"-c", "--create"}, description = "create entries before any deletes")
+	    boolean createEntries = false;
+	    @Parameter(names = {"-r", "--read"}, description = "read entries after any creates/deletes")
+	    boolean readEntries = false;
+	    
+	    public Opts() {
+	      super(DEFAULT_TABLE_NAME);
+	      auths = new Authorizations();
+	    }
+	  }
+	
+	public void enableTracing(Opts opts) throws Exception {
+		DistributedTrace.enable(opts.getInstance(), new ZooReader(opts.getInstance().getZooKeepers(),
1000), "myHost", "myApp");
+	}
+	
+	public void execute(Opts opts) throws TableNotFoundException, InterruptedException, AccumuloException,
AccumuloSecurityException, TableExistsException {
+		
+	    if (opts.createtable) {
+	    	opts.getConnector().tableOperations().create(opts.getTableName());
+	    }
+	    
+	    if (opts.createEntries) {
+	    	createEntries(opts);
+	    }
+	    
+	    if (opts.readEntries) {
+	    	readEntries(opts);
+	    }
+		
+	    if (opts.deletetable) {
+	    	opts.getConnector().tableOperations().delete(opts.getTableName());
+	    }
+	}
+	
+	private void createEntries(Opts opts) throws TableNotFoundException, AccumuloException,
AccumuloSecurityException {
+		
+		
+		BatchWriter batchWriter = opts.getConnector().createBatchWriter(opts.getTableName(), new
BatchWriterConfig());
+	    
+		Mutation m = new Mutation("row");
+		m.put("cf", "cq", "value");
+		
+		// Trace the write operation. Note, unless you flush the BatchWriter, you will not capture
+		// the write operation as it is occurs asynchronously. You can optionally create additional
Spans
+		// within a given Trace as seen below around the flush
+		Trace.on("Client Write");
+		
+	    batchWriter.addMutation(m);
+	    Span flushSpan = Trace.start("Client Flush");
+	    batchWriter.flush();
+	    flushSpan.stop();
+	    
+	    // Use Trace.offNoFlush() if you don't want the operation to block.
+	    Trace.off();
+	    
+	    batchWriter.close();
+	}
+	
+	private void readEntries(Opts opts) throws TableNotFoundException, AccumuloException, AccumuloSecurityException
{
+		
+		Scanner scanner = opts.getConnector().createScanner(opts.getTableName(), opts.auths);
+		
+		// Trace the read operation.  
+		Span readSpan = Trace.on("Client Read");
+		
+		int numberOfEntriesRead = 0;
+		for (Entry<Key,Value> entry : scanner) {
+			System.out.println(entry.getKey().toString() + " -> " + entry.getValue().toString());
+			++numberOfEntriesRead;
+	    }
+		// You can add additional metadata (key, values) to Spans which will be able to be viewed
in the Monitor
+		readSpan.data("Number of Entries Read", String.valueOf(numberOfEntriesRead));
+		
+		Trace.off();
+	}
+	
+	
+	/**
+	 * @param args
+	 * @throws AccumuloSecurityException 
+	 * @throws AccumuloException 
+	 * @throws TableNotFoundException 
+	 * @throws InterruptedException 
+	 * @throws KeeperException 
+	 * @throws IOException 
+	 * @throws TableExistsException 
+	 */
+	public static void main(String[] args) throws Exception {
+	
+		TracingExample tracingExample = new TracingExample();
+		Opts opts = new Opts();
+		ScannerOpts scannerOpts = new ScannerOpts();
+		opts.parseArgs(TracingExample.class.getName(), args, scannerOpts);
+		
+		tracingExample.enableTracing(opts);
+		tracingExample.execute(opts);
+	}
+
+}


Mime
View raw message