htrace-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject incubator-htrace git commit: HTRACE-350. Add a developer guide (Colin Patrick McCabe via iwasakims)
Date Tue, 05 Apr 2016 16:43:31 GMT
Repository: incubator-htrace
Updated Branches:
  refs/heads/master 45e69c888 -> 51dbeb835

HTRACE-350. Add a developer guide (Colin Patrick McCabe via iwasakims)


Branch: refs/heads/master
Commit: 51dbeb8355b856cf27cf5d42d13ee75a7c53c2c5
Parents: 45e69c8
Author: Masatake Iwasaki <>
Authored: Wed Apr 6 01:42:44 2016 +0900
Committer: Masatake Iwasaki <>
Committed: Wed Apr 6 01:42:44 2016 +0900

----------------------------------------------------------------------                                 |  13 +-
 src/main/site/markdown/ | 396 +++++++++++++++++++++++++
 src/main/site/markdown/           | 141 ---------
 src/main/site/site.xml                    |   2 +-
 4 files changed, 406 insertions(+), 146 deletions(-)
diff --git a/ b/
index 39a2331..7b11dab 100644
--- a/
+++ b/
@@ -17,8 +17,13 @@
-HTrace is a tracing framework for use with distributed systems.  See the
-documentation at [src/main/site/markdown/] or at
- for more details.
+[HTrace]( is a tracing framework for use
+with distributed systems.
-For help on building the software, see [BUILDING.txt].
+For help on building the software, see [BUILDING.txt](./BUILDING.txt).
+For details about integrating HTrace into an application, see the [developer
+See the [release guide](./src/main/site/markdown/ for
+information about making an HTrace release.
diff --git a/src/main/site/markdown/ b/src/main/site/markdown/
new file mode 100644
index 0000000..20b832b
--- /dev/null
+++ b/src/main/site/markdown/
@@ -0,0 +1,396 @@
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+# HTrace Developer Guide
+## Introduction
+Apache HTrace is an open source framework for distributed tracing.  It can be
+used with both standalone applications and libraries.
+By adding HTrace support to your project, you will allow end-users to trace
+their requests.  In addition, any other project that uses HTrace can follow the
+requests it makes to your project.  That`s why we say HTrace is "end-to-end."
+HTrace was designed for use in big distributed systems such as the Apache
+Hadoop Distributed Filesystem and the Apache HBase storage engine.  However,
+there is nothing Hadoop-specific about HTrace.  It has no dependencies on
+Hadoop, and is a useful building block for many distributed systems.
+## The HTrace Core Library
+In order to use HTrace, your application must link against the appropriate core
+library.  HTrace`s core libraries have been carefully designed to minimize the
+number of dependencies that each one pulls in.  HTrace currently has Java, C,
+and C++ support.
+HTrace guarantees that the API of core libraries will not change in an
+incompatible way during a minor release.  So if your application worked with
+HTrace 4.1, it should continue working with HTrace 4.2 with no code changes.
+(However HTrace 5 may change things, since it is a major release.)
+### Java
+The Java library for HTrace is named htrace-core4.jar.  This jar must appear on
+your CLASSPATH  in order to use tracing in Java.  If you are using Maven, add
+the following to your dependencyManagement section:
+    <dependencyManagement>
+      <dependencies>
+        <dependency>
+          <groupId>org.apache.htrace</groupId>
+          <artifactId>htrace-core4</artifactId>
+          <version>4.1.0-incubating</version>
+        </dependency>
+        ...
+      </dependencies>
+      ...
+    </dependencyManagement>
+If you are using an alternate build system, use the appropriate configuration
+for your build system.  Note that it is not a good idea to shade htrace-core4,
+because some parts of the code use reflection to load classes by name.
+### C
+The C library for HTrace is named  The interface for
+is described in [htrace.h](
+As with all dynamically loaded native libraries, your application or library
+must be able to locate in order to use it.  There are many ways to
+accomplish this.  The easiest way is to put in one of the system
+shared library paths.  You can also use RPATH or LD\_LIBRARY\_PATH to alter the
+search paths which the operating system uses.
+### C++
+The C++ API for HTrace is a wrapper around the C API.  This approach makes it
+easy to use HTrace with any dialect of C++ without recompiling the core
+library.  It also makes it easier for us to avoid making incompatible ABI
+The interface is described in
+the same as using the C API, except that you use htrace.hpp instead of
+## Core Concepts
+HTrace is based on a few core concepts.
+### Spans
+in HTrace are lengths of time.  A span has a beginning time in milliseconds, an
+end time, a description, and many other fields besides.
+Spans have parents and children.  The parent-child relationship between spans
+is a little bit like a stack trace.  For example, the span graph of an HDFS
+"ls" request might look like this:
+    ls
+    +--- FileSystem#createFileSystem
+    +--- Globber#glob
+    |  +---- GetFileInfo
+    |      +---- ClientNamenodeProtocol#GetFileInfo
+    |          +---- ClientProtocol#GetFileInfo
+    +--- listPaths
+       +---- ClientNamenodeProtocol#getListing
+           +---- ClientProtocol#getListing
+"ls" has several children, "FileSystem#createFileSystem", "Globber#glob", and
+"listPaths".  Those spans, in turn, have their own children.
+Unlike in a traditional stack trace, the spans in HTrace may be in different
+processes or even on different computers.  For example,
+ClientProtocol#getListing is done on the NameNode, whereas
+ClientNamenodeProtocol#getListing happens inside the HDFS client.  These are
+usually on different computers.
+Each span has a unique 128-bit ID.  Because the space of 128-bit numbers is so
+large, HTrace can use random generation to avoid collisions.
+### Scopes
+objects manage the lifespan of Span objects.  When a TraceScope is created, it
+often comes with an associated Span object.  When this scope is closed, the
+Span will be closed as well.  "Closing" the scope means that the span is sent
+to a SpanReceiver for processing.  We will talk more about what that means
+later.  For now, just think of closing a TraceScope as similar to closing a
+file descriptor-- the natural thing to do when the TraceScope is done.
+HTrace tracks whether a trace scope is active in the current thread by using
+thread-local data.  This approach makes it easier to add HTrace to existing
+code, by avoiding the need to pass around context objects.
+TraceScopes lend themselves to the try... finally pattern of management:
+    TraceScope computationScope = tracer.newScope("CalculateFoo");
+    try {
+        calculateFoo();
+    } finally {
+        computationScope.close();
+    }
+Any trace spans created inside calculateFoo will automatically have the
+CalculateFoo trace span we have created here as their parents.  We don`t have
+to do any additional work to set up the parent/child relationship because the
+thread-local data takes care of it.
+In Java7, the
+idiom may be used to accomplish the same thing without a finally block:
+    try (TraceScope computationScope = tracer.newScope("CalculateFoo")) {
+        calculateFoo();
+    }
+The important thing to remember is to close the scope when you are done with it.
+Note that in the C++ API, the destructor of the htrace::Scope object will
+automatically close the span.
+    htrace::Scope(tracer_, "CalculateFoo");
+    calculateFoo();
+TraceScope are associatewith particular threads.  If you want to pass a trace
+scope to another thread, you must detach it from the current one first.  We
+will talk more about that later in this guide.
+### Tracers
+are the API for creating trace scope objects.  You can see that in the example
+above, we called the Tracer#newScope function to create a scope.
+It is difficult to trace every operation.  The volume of trace span data that
+would be generated would be extremely large!  So we rely on sampling a subset
+of all possible traces.  Tracer objects contain Samplers.  When you call
+Tracer#newScope, the Tracer will consult that Sampler to determine if a new
+span should be created, or if an empty scope which contains no span should be
+returned.  Note that if there is already a currently active span, the Tracer
+will always create a child span, regardless of what the sampler says.  This is
+because we want to see the complete graph of every operation, not just "bits
+and pieces." Tracer objects also manage the SpanReceiver objects which control
+where spans are sent.
+A single process or library can have many Tracer objects.  Each Tracer object
+has its own configuration.  One way of thinking of Tracer objects is that they
+are similar to Log objects in log4j.  Just as you might create a Log object for
+the NameNode and one for the DataNode, we create a Tracer for the NameNode and
+another Tracer for the DataNode.  This allows users to control the sampling
+rate for the DataNode and the NameNode separately.
+Unlike TraceScope and Span, Tracer objects are thread-safe.  It is perfectly
+acceptable (and even recommended) to have multiple threads calling
+Tracer#newScope at once on the same Tracer object.
+The number of Tracer objects you should create in your project depends on the
+structure of your project.  Many applications end up creating a small number of
+global Tracer objects.  Libraries usually should not use globals, but associate
+the Tracer with the current library context.
+### Wrappers
+HTrace contains many helpful wrapper objects like
+These helper classes make it easier for you to create trace spans.  Basically,
+they act as wrappers around Tracer#newScope.
+## SpanReceivers
+HTrace is a pluggable framework.  We can configure where trace spans are sent
+at runtime, by selecting the appropriate SpanReceiver.
+    FoobarApplication
+          |
+          V
+    htrace-core4
+          |
+          V
+    HTracedSpanReceiver
+         OR
+    LocalFileSpanReceiver
+         OR
+    StandardOutSpanReceiver
+         OR
+    ZipkinSpanReceiver
+         OR
+         ...
+As a developer integrating HTrace into your application, you do not need to
+know what each and every SpanReceiver does-- only the ones you actually intend
+to use.   The nice thing is that users can use any span receiver with your
+project, without any additional effort on your part.  The span receivers are
+decoupled from the core library.
+When using Java, you will need to add the jar file for whichever span receiver
+you want to use to your CLASSPATH.  (These span receivers are not added to the
+CLASSPATH by default because they may have additional dependencies that not
+every user wants.) For C and C++, a more limited set of span receivers is
+available, but they are all integrated into, so no additional
+libraries are needed.
+## Configuration
+Clearly, HTrace requires configuration.  We need to control which SpanReceiver
+is used, what the sampling rate is, and many other things besides.  Luckily, as
+we discussed earlier, the Tracer objects maintain this configuration
+information for us.  When we ask for a new trace scope, the Tracer knows what
+configuration to use.
+This configuration comes from the HTraceConfiguration object that we supplied
+to the Tracer#Builder earlier.  In general, we want to configure HTrace the
+same way we configure anything else in our distributed system.  So we normally
+create a subclass of HTraceConfiguration that accesses the appropriate
+information in our existing configuration system.
+To make this a little more concrete, let`s suppose we are writing Bob`s
+Distributed System.  Being a pragmatic (not to mention lazy) guy, Bob has
+decided to just use Java configuration properties for configuration.
+So our Tracer#Builder invoation would look something like this:
+    this.tracer = new Tracer.Builder("Bob").
+        conf(new HTraceConfiguration() {
+            @Override
+            public String get(String key) {
+              return System.getProperty("htrace." + key);
+            }
+            @Override
+            public String get(String key, String defaultValue) {
+              String ret = get(key);
+              return (ret != null) ? ret : defaultValue;
+            }
+        }).
+        build();
+You can see that this configuration object maps every property starting in
+"htrace." to an htrace property.  So, for example, you would set the Java
+system property value "htrace.span.receiver.classes" in order to control the
+HTrace configuration key "span.receiver.classes".
+Of course, Bob probably should have been less lazy, and used a real
+configuration system instead of using Java system properties.  This is just a
+toy example to illustrate how to integrate with an existing configuration
+Bob might also have wanted to use different prefixes for different Tracer
+objects.  For example, in Hadoop you can configure the FsShell Tracer
+separately from the NameNode Tracer, by setting
+"".  This is easy to control by changing
+the HTraceConfiguration object that you pass to Tracer#Builder.
+Note that in C and C++, this system is a little different, based on explicitly
+creating a configuration object prior to creating a tracer, rather than using a
+callback-based system.
+## TracerPool
+SpanReceiver objects often need to make a network connection to a remote
+serveice or daemon.  Usually, we don`t want to create more than one
+SpanReceiver of each type in a particular process, so that we can optimize the
+number of these connections that we have open.  TracerPool objects allow us to
+acheieve this.
+Each Tracer object belongs to a TracerPool.  When a call to Tracer#Builder is
+made which requests a specific SpanReceiver, we check the TracerPool to see if
+there is already an instance of that SpanReceiver.  If so, we simply re-use the
+existing one rather than creating a new one.
+Normally, you don`t need to worry about TracerPools.  However, if you have an
+explicit need to create multiple SpanReceivers of the same type, you can do it
+by using a TracerPool other than the default one, or by explicitly adding the
+SpanReceiver to your Tracer once it has been created.  This is not a very
+common need.
+When the application terminates, we will attempt to close all currently open
+SpanReceivers.  You can attempt to close the SpanReceivers earlier than that by
+calling tracer.getTracerPool().removeAndCloseAllSpanReceivers().
+## Passing Span IDs over RPC
+So far, we have described how to use HTrace inside a single process.  However,
+since we are dealing with distributed systems, a single process is not enough.
+We need a way to send HTrace information across the network.
+Unlike some other tracing systems, HTrace works with many different RPC
+systems.  You do not need to change the RPC framework you are using in order to
+use HTrace.  You simply need to find a way to pass HTrace information using the
+RPC framework that you`re already using.  In most cases, what this boils down
+to is figuring out a way to send the 128-bit parent ID of an operation over the
+network as an optional field.
+Let`s say that Bob is writing the server side of his system.  If the client
+sent its parent ID over the wire, Bob might write code like this:
+    BobRequestProto bp = ...;
+    SpanId spanId = (bp.hasSpanId()) ? bp.getSpanId() : SpanId.INVALID;
+    try (TraceScope scope = tracer.newScope("bobRequest", spanId)) {
+        doBobRequest(bp);
+    }
+By passing the spanId to Tracer#newScope, we ensure that any new span we create
+will have a record of its parent.
+## Handling work done in multiple threads
+Sometimes, you end up performing work for a single request in multiple threads.
+How can we handle this?  Certainly, we can use the same approach as we did in
+the RPC case above.  We can have the child threads create trace scopes which
+use our parent ID object.  SpanId objects are immtuable and easy to share
+between threads.
+    try (TraceScope bigScope = tracer.newScope("bigComputation")) {
+        SpanId bigScopeId = bigScope.getCurrentSpanId();
+        Thread t1 = new Thread(new Runnable() {
+            @Override
+            public void run() {
+                TraceScope scope = (bigScopeId.isValid()) ?
+                    tracer.newScope("bigComputationWorker", bigScopeId) :
+                    tracer.newNullScope();
+                try {
+                    doWorkerStuff();
+                } finally {
+                    scope.close();
+                }
+            }
+        }, "myWorker");
+        t1.start();
+        t1.join();
+    }
+Note that in this case, the two threads are not sharing trace scopes.  Instead,
+we are setting up a new trace scope, which may have its own span, which has the
+outer scope as a parent.  Note that HTrace will be well-behaved even if the
+outer scope may be closed before the inner one.  The SpanId object of a
+TraceScope continues to be valid even after a scope is closed.  It`s just a
+number, essentially-- and that number will not be reused by any other scope.
+Why do we make the worker Thread call newNullScope in the case where the outer
+scope`s span id is invalid?  Well, we don`t want to ever create the inner span
+if the outer one does not exist.  Calling newNullScope ensures that we get a
+scope with no span, no matter what samplers are configured.
+What if we don`t want to create more than one span here?  In that case, we need
+to have some way of detaching the TraceScope from the parent thread, and
+re-attaching it to the worker thread.  Luckily, HTrace has an API for that.
+    final TraceScope bigScope = tracer.newScope("bigComputation");
+    bigScope.detach();
+    Thread t1 = new Thread(new Runnable() {
+        @Override
+        public void run() {
+            bigScope.reattach();
+            try {
+                doWorkerStuff();
+            } finally {
+                bigScope.close();
+            }
+        }
+    }, "myWorker");
+    t1.start();
+    t1.join();
+Note that in this case, we don`t need to close the TraceScope in the containing
+thread.  It has already been closed by the worker thread.
diff --git a/src/main/site/markdown/ b/src/main/site/markdown/
deleted file mode 100644
index 4db4244..0000000
--- a/src/main/site/markdown/
+++ /dev/null
@@ -1,141 +0,0 @@
-  Licensed under the Apache License, Version 2.0 (the "License");
-  you may not use this file except in compliance with the License.
-  You may obtain a copy of the License at
-  Unless required by applicable law or agreed to in writing, software
-  distributed under the License is distributed on an "AS IS" BASIS,
-  See the License for the specific language governing permissions and
-  limitations under the License. See accompanying LICENSE file.
-Apache HTrace is an <a href="">Apache Incubator</a>
-project. To add HTrace to your project, see detail on how to add it as a
-<a href="dependency-info.html">dependency</a>. Formerly, HTrace was available
at `org.htrace`.
-* htrace-4.0.0-incubating published September 15th, 2015. [Download it!](
-* htrace-3.2.0-incubating published June 2nd, 2015. [Download it!](
-* We made our first release from Apache Incubator, htrace-3.1.0-incubating, January 20th,
2015. [Download it!](
-## API
-Using HTrace requires adding some instrumentation to your application.
-Before we get into the details, lets review our terminology.  HTrace
-borrows [Dapper's](
-<b>Span:</b> The basic unit of work. For example, sending an RPC is a new span,
-as is sending a response to an RPC.  Spans are identified by a unique 128-bit
-ID.  Spans also have other data, such as descriptions, key-value annotations,
-the ID of the span that caused them, and tracer IDs.
-Spans are started and stopped, and they keep track of their timing
-information.  Once you create a span, you must stop it at some point
-in the future.
-<b>TracerId:</b> Identifies the Tracer object which created a specific Span.
-The TracerId should identify the source of the Span.  "" is a
-typical TracerID.
-<b>SpanReceiver:</b> SpanReceivers handle spans once they have been created.
-Typically, Span Receivers send spans to an external data store to be
-analyzed.  HTrace comes with several standard span receivers, such as
-<b>Sampler:</b> Samplers determine when tracing should be enabled, and when it
-should be disabled.   The goal of HTrace is to trace an entire request.
-## How to add tracing to your application
-To instrument your system you must:
-### Create a Tracer object
-You can create a Tracer object via the `Tracer#Builder`.
-    Tracer tracer = new Tracer.Builder("MyApp").conf(conf).build();
-The `Tracer#Builder` will take care of creating the appropriate Sampler and
-SpanReceiver objects, as well as the Tracer itself.   If a SpanReceiver was
-created, we will install a shutdown hook to close it when the JVM shuts down.
-### Attach additional information to your RPCs
-In order to create the causal links necessary for a trace, HTrace needs to know
-about the causal relationships between spans.  The only information you need to
-add to your RPCs is the 128-bit span ID.  If tracing is enabled when you send an
-RPC, attach the ID of the current span to the message.  On the receiving end of
-the RPC, check to see if the message has a span ID attached.  If it does, start
-a new trace scope with that span as a parent.
-### Wrap your thread changes
-HTrace stores span information in java's ThreadLocals, which causes
-the trace to be "lost" on thread changes. The only way to prevent
-this is to "wrap" your thread changes. For example, if your code looks
-like this:
-    Thread t1 = new Thread(new MyRunnable());
-    ...
-Just change it to look this:
-    Thread t1 = new Thread(tracer.wrap(new MyRunnable(), "MyRunnable"));
-That's it! `Tracer.wrap()` takes two arguments
-(a runnable or a callable and span description)
-and if the current thread is a part of a trace,
-returns a wrapped version of the runnable/callable.
-The wrapped version of a runnable/callable
-just knows about the span that created it and will start a new span
-with given description in the new thread that is the child of the span that
-created the runnable/callable.  There may be situations in which a
-simple `Tracer.wrap()` does not suffice.  In these cases all you need
-to do is keep a reference to the "parent span" (the span before the
-thread change) and once you're in the new thread start a new span that
-is the "child" of the parent span you stored.
-For example:
-Say you have some object representing a "put" operation.  When the
-client does a "put," the put is first added to a list so another
-thread can batch together the puts. In this situation, you
-might want to add another field to the Put class that could store the
-current span at the time the put was created.  Then when the put is
-pulled out of the list to be processed, you can start a new span as
-the child of the span stored in the Put.
-### Add custom spans and annotations
-Once you've augmented your RPC's and wrapped the necessary thread
-changes, you can add more spans and annotations wherever you want.
-For example, you might do some expensive computation that you want to
-see on your traces.  In this case,
-you could create a new trace scope which starts a new span internally
-before the computation that you then close after the computation has
-finished. It might look like this:
-    TraceScope computationScope = tracer.newScope("Expensive computation.");
-    try {
-        //expensive computation here
-    } finally {
-        computationScope.close();
-    }
-HTrace also supports key-value annotations on a per-trace basis.
-    scope.addKVAnnotation("faultyRecordCounter", Integer.toString(1));
-## htrace-zipkin
-htrace-zipkin provides the `SpanReceiver` implementation
-which sends spans to [Zipkin]( collector.
-You can build the uber-jar (htrace-zipkin-*-jar-withdependency.jar) for manual
-setup as shown below.  This uber-jar contains all dependencies except
-htrace-core and its dependencies.
-    $ cd htrace-zipkin
-    $ mvn compile assembly:single
-## htrace-hbase
-See htrace-hbase for an Span Receiver implementation that writes HBase.
diff --git a/src/main/site/site.xml b/src/main/site/site.xml
index 544db38..f2b329c 100644
--- a/src/main/site/site.xml
+++ b/src/main/site/site.xml
@@ -33,7 +33,7 @@
     <menu name="Apache HTrace Project">
-      <item name="Overview" href="index.html"/>
+      <item name="Developer Guide" href="developer_guide.html"/>
       <item name="License" href="license.html" />
       <item name="Downloads" href=""
       <item name="Distribution" href="dependency-management.html" />

View raw message