htrace-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin P. McCabe" <>
Subject Re: HTrace for Nutch 2.x Search Stack
Date Fri, 27 Feb 2015 19:58:54 GMT
Hi Lewis,

Good questions.  I would say HTrace differs from TRACE logging (or
other single-node metrics, JMX, audit logs, etc.) in that it pulls
together information from across the cluster.  This is something that
is a major pain point when using a distributed system such as HDFS.
Just to diagnose a slow write, you might have to match up logs from a
client log and the logs of 3 different datanodes.  The big idea behind
htrace is two things: integrating those logging sources, and using
sampling to instrument performance in production.  The main thing
htrace deals with is "spans" which are lengths of time.

We're working on a web UI that will allow people to search for spans
by time, duration, and name (among other things).  It's not quite
finished now (hoping to have something usable in HTrace 3.2.0 or maybe
3.3.0... but abe can comment more on that.)

Here's an early screenshot (probably way out of date now):

There is also a plan to create a visualization of parent/child
relationships on the web UI, by using the d3 library (which can draw
graphs, and do many other things besides.)

In the meantime, there's an option to product a graphviz file from a
file containing span JSON.  That way you can draw a graph of
parent/child relationships with the "dot" tool, available on Linux.
Uh... unfortunately it's broken right now... let me file a JIRA for
that :P  This is a very new feature, got added earlier this week.

The web UI is a great place to get involved right now... there is a
lot of work going on there and we've been adding new contributors.


On Thu, Feb 26, 2015 at 1:46 PM, Lewis John Mcgibbney
<> wrote:
> Hi Nick,
> Grand. Thank you
> What is visualization looking like right now? It there currently a
> mechanism for visualizing HTrace structures?
> Is it worth considering posting something like this as a GSoC project is
> one does not currently exist?
> Thanks
> Lewis
> On Thu, Feb 26, 2015 at 1:31 PM, Nick Dimiduk <> wrote:
>> Hi Lewis,

View raw message