-

Overview

Apache HTrace is a tracing framework intended for use with distributed systems written in java. Since version 3.1.3, Apache CXF fully supports integration with Apache HTrace, both on client side and server side. This section gives a complete overview on how distributed tracing support is supported in JAX-RS applications built on top of Apache CXF.

Distributed Tracing in Nutshell

Distributed tracing, first described by Google in Dapper, a Large-Scale Distributed Systems Tracing Infrastructure paper became increasingly important topic these days. With microservices (aka SOA) gaining more and more adoption, the typical applications are built using dozens or even hundreds of small, distributed pieces. The end-to-end traceability of the requests (or any kind of work performed on user's behalf) is hard task to accomplish, particularly taking into account asyncronous or/and concurrent invocations. Apache HTrace is inspired by Dapper, a Large-Scale Distributed Systems Tracing Infrastructure paper and essentially is a full-fledged distributed tracing framework.

Distributed tracing is additional instrumentation layer on top of new or existing applications. In terms of distributed tracing, span represents a basic unit of work. For example, executing database query is a span. Spans are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span is a part of. Spans also have other data, such as descriptions, timelines, key-value annotations, the ID of the span that caused them (parent), and process ID’s (normally IP address and process name). Spans are started and stopped, and they keep track of their timing information. Once span is created, it should be stopped at some point in the future. In turn, trace is a set of spans forming a tree-like structure. For example, if you are running a JAX-RS service, a trace might be formed by a PUT request.

From implementation prospective, and in context of Java applications, spans are attached to their threads (in general, thread which created the span should close it). However it is possible to transfer spans from thread to thread in order to model a complex execution flows. It is also possible to have many spans in the same thread, as long as they are properly created and closed. In the next sections we are going to see the examples of that.

Another two important concepts of in context of distributed tracing are span receivers and samplers. Essentially, all spans (including start/stop time, key/value annotations, timelines, ..) should be persisted (or collected) somewhere. Span receiver is a collector within a process that is the destination of spans when a trace is running (it could be a console, local file, data store, ...). Apache HTrace provides span receivers for Apache HBase, Apache Flume and Twitter Zipkin. From other side, samplers allow to control the frequency of the tracing (all the time, never, probability driven, ...). Using the sampler is the way to minimize tracing overhead (or just amount of traces) by limiting them to particular conditions.

Distributed Tracing in Apache CXF

Apache CXF is a very popular framework for building services and web APIs. No doubts, it is going to play even more important role in context of microservices architecture letting developers to quickly build and deploy individual JAX-RS/JAX-WS services. As it was just mentioned before, distributed tracing is an essential tec hnique to monitor the application as whole, breaking the request to individual service traces as it goes through and crosses the boundaries of threads, processes and machines.

The current integration of distributed tracing in Apache CXF supports Apache HTrace only in JAX-RS 2.x applications. From high-level prospective, it consists of three main parts:

TracerContext (injectable through @Context annotation)
HTraceProvider (server-side JAX-RS provider) and HTraceClientProvider (client-side JAX-RS provider)
HTraceFeature (server-side Apache CXF feature to simplify Apache HTrace configuration and integration)

Apache CXF uses HTTP headers to hand off tracing context from the client to the service and from the service to service. Those headers are used internally by HTraceProvider and HTraceClientProvider, but are configurable. The default header names are declared in the TracerHeaders class:

X-Trace-Id: contains a current trace ID
X-Span-Id: contains a current span ID

By default, HTraceProvider will try pass the currently active span through HTTP headers on each service invocation. If there is no active span, the new span will be created and passed through HTTP headers on per-invocation basis. Essentially, just registering the HTraceProvider on the client and HTraceClientProvider on the server is enough to have tracing context to be properly passed everywhere. The only configuration part which is necessary are span receiver(s) and sampler.

Configuring Client

There are a couple of way the JAX-RS client could be configured, depending on the client implementation. Apache CXF provides its own WebClient which could be configured just like that (in future versions, there would be a simpler ways to do that using client specific features):

+/*]]>*/
+
Overview
Distributed Tracing in Nutshell
Distributed Tracing in Apache CXF
Configuring Client
Configuring Server
Distributed Tracing In Action: Usage Scenarios
+
Overview
Apache HTrace is a tracing framework intended for use with distributed systems written in java. Since version 3.1.3, Apache CXF fully supports integration with Apache HTrace, both on client side and server side. This section gives a complete overview on how distributed tracing support is supported in JAX-RS applications built on top of Apache CXF.
Distributed Tracing in Nutshell
Distributed tracing, first described by Google in Dapper, a Large-Scale Distributed Systems Tracing Infrastructure paper became increasingly important topic these days. With microservices (aka SOA) gaining more and more adoption, the typical applications are built using dozens or even hundreds of small, distributed pieces. The end-to-end traceability of the requests (or any kind of work performed on user's behalf) is hard task to accomplish, particularly taking into account asyncronous or/and concurrent invocations. Apache HTrace is inspired by Dapper, a Large-Scale Distributed Systems Tracing Infrastructure paper and essentially is a full-fledged distributed tracing framework.
Distributed tracing is additional instrumentation layer on top of new or existing applications. In terms of distributed tracing, span represents a basic unit of work. For example, executing database query is a span. Spans are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span is a part of. Spans also have other data, such as descriptions, timelines, key-value annotations, the ID of the span that caused them (parent), and process ID’s (normally IP address and process name). Spans are started and stopped, and they keep track of their timing information. Once span is created, it should be stopped at some point in the future. In turn, trace is a set of spans forming a tree-like structure. For example, if you are running a JAX-RS service, a trace might be formed by a PUT request.
From implementation prospective, and in context of Java applications, spans are attached to their threads (in general, thread which created the span should close it). However it is possible to transfer spans from thread to thread in order to model a complex execution flows. It is also possible to have many spans in the same thread, as long as they are properly created and closed. In the next sections we are going to see the examples of that.
Another two important concepts of in context of distributed tracing are span receivers and samplers. Essentially, all spans (including start/stop time, key/value annotations, timelines, ..) should be persisted (or collected) somewhere. Span receiver is a collector within a process that is the destination of spans when a trace is running (it could be a console, local file, data store, ...). Apache HTrace provides span receivers for Apache HBase, Apache Flume and Twitter Zipkin. From other side, samplers allow to control the frequency of the tracing (all the time, never, probability driven, ...). Using the sampler is the way to minimize tracing overhead (or just amount of traces) by limiting them to particular conditions.
Distributed Tracing in Apache CXF
Apache CXF is a very popular framework for building services and web APIs. No doubts, it is going to play even more important role in context of microservices architecture letting developers to quickly build and deploy individual JAX-RS/JAX-WS services. As it was just mentioned before, distributed tracing is an essential tec hnique to monitor the application as whole, breaking the request to individual service traces as it goes through and crosses the boundaries of threads, processes and machines.
The current integration of distributed tracing in Apache CXF supports Apache HTrace only in JAX-RS 2.x applications. From high-level prospective, it consists of three main parts:
TracerContext (injectable through @Context annotation)
HTraceProvider (server-side JAX-RS provider) and HTraceClientProvider (client-side JAX-RS provider)
HTraceFeature (server-side Apache CXF feature to simplify Apache HTrace configuration and integration)
Apache CXF uses HTTP headers to hand off tracing context from the client to the service and from the service to service. Those headers are used internally by HTraceProvider and HTraceClientProvider, but are configurable. The default header names are declared in the TracerHeaders class:
X-Trace-Id: contains a current trace ID
X-Span-Id: contains a current span ID
By default, HTraceProvider will try pass the currently active span through HTTP headers on each service invocation. If there is no active span, the new span will be created and passed through HTTP headers on per-invocation basis. Essentially, just registering the HTraceProvider on the client and HTraceClientProvider on the server is enough to have tracing context to be properly passed everywhere. The only configuration part which is necessary are span receiver(s) and sampler.
Configuring Client
There are a couple of way the JAX-RS client could be configured, depending on the client implementation. Apache CXF provides its own WebClient which could be configured just like that (in future versions, there would be a simpler ways to do that using client specific features):

final Map<String, String> properties = new HashMap<String, String>(); final HTraceConfiguration conf = HTraceConfiguration.fromMap(properties); Trace.addReceiver(new StandardOutSpanReceiver(conf)); @@ -146,7 +146,7 @@ final Response response = client .request() .accept(MediaType.APPLICATION_JSON) .get();
-
Configuring Server
Server configuration is a bit simpler than client one thanks to the feature class available, HTraceFeature. Depending on the way the Apache CXF is used to configure JAX-RS services, it could be part of JAX-RS application configuration, for example:
+
Configuring Server
Server configuration is a bit simpler than client one thanks to the feature class available, HTraceFeature. Depending on the way the Apache CXF is used to configure JAX-RS services, it could be part of JAX-RS application configuration, for example:

@ApplicationPath("/") public class CatalogApplication extends Application { @Override @@ -171,7 +171,7 @@ final JAXRSServerFactoryBean factory = R factory.setFeatures(Arrays.< Feature >asList(new HTraceFeature(HTraceConfiguration.fromMap(properties)))); ... return factory.create();
-
Once the span receiver(s) and sampler are properly configured, all generated spans are going to be collected and available for analysis and/or visualization.

+

Once the span receiver(s) and sampler are properly configured, all generated spans are going to be collected and available for analysis and/or visualization.

Distributed Tracing In Action: Usage Scenarios

In the following subsections we are going to walk through many different scenarios to illustrate the distributed tracing in action, starting from the simplest ones and finishing with asynchronous JAX-RS services. All examples assume that configuration has been done (see please Configuring Client and Configuring Server sections above).