htrace-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Wall <mjw...@gmail.com>
Subject Re: HTrace and Go
Date Sun, 01 Mar 2015 16:58:36 GMT
Thanks Colin, that helps a lot.

Couple more.

1)  In htrace-flume, htrace-hbase, and htrace-zipkin, I don't see any
mention of starting up the htraced daemon.  Do these collectors use htraced
currently?  Trying to wrap my head around how Go will do that.
2)  Are the REST endpoints documented anywhere?  I found
https://issues.apache.org/jira/browse/HTRACE-105.  From htraced/rest.go, I
found the following
    278: r.Handle("/server/info", &serverInfoHandler{lg:
rsv.lg}).Methods("GET")
    282: r.Handle("/writeSpans", writeSpansH).Methods("POST")
    285: r.Handle("/query", queryH).Methods("GET")
Anything else currently planned?  Will
https://issues.apache.org/jira/browse/HTRACE-118 need one?
3) There is a web UI discussed in htrace-hbase.  Will that be replaced by
the htraced UI?  The spans.png and timelines.png in that directory are
really what I am looking for.
4)  Is there a way to correlate traces after the fact?  Say I have an
ingest pipeline that processes data then writes it to HDFS.  Then another
process picks it up from HDFS and makes it available for query.  I'd like
to take spans from both parts and tie them together so I can trace
something all the way through.  Any thoughts on that?

Thanks again

Mike

On Sat, Feb 28, 2015 at 11:45 PM, Colin P. McCabe <cmccabe@apache.org>
wrote:

> Thanks for this Mike!  It should be helpful.  I've been planning on
> doing a blog post about getting htraced set up, but there are a few
> bugs right now that I'm trying to fix first.
>
> Good writeup.  One note is that I think most Linux distros newer than
> CentOS 6 will be able to do "yum install leveldb-devel" or similar
> (maybe even CentOS 7?)  Also, brew is a nice option on MacOS if you
> are already using that.
>
> One other note is that we automatically install the "godep" program.
> So it should not be necessary to install that manually.  The
> BUILDING.txt is wrong here (it says that godep needs to be installed
> by hand.)  I have filed HTRACE-126 to correct this part of
> BUILDING.txt.
>
> > 1)  Is REST the only interface for getting span info in and out of
> htraced?
>
> Yes.  I wrote some code to do a protobuf interface, but I haven't
> posted it yet.  I think we should stabilize the other stuff before
> going for the extra performance that a non-REST interface would give.
>
> > 2)  How is HTracedRESTReciever in htrace-htraced/ used or how will it be
> used?  I didn't see it anywhere in the Java clients.
>
> Span Receivers are basically "sinks" where spans can go.  Client code,
> such as Hadoop or HBase, doesn't need to care about which span
> receiver the spans are going to.  But you will need to put the
> htrace-htraced jar on your CLASSPATH to use it.
>
> > 3)  Is it expected that there will be one htraced process running.  I am
> thinking of it like the collector in
> > Zipkin, http://twitter.github.io/zipkin/Architecture.html.  Is that
> correct?  Is there a similar architecture
> > diagram for HTrace to help me see the big picture?  Happy to help make
> one once I figure it out.
>
> Yes, that's right.  There is currently one htraced per cluster.  Maybe
> later if load becomes a problem we'll shard it.  You are correct that
> it is similar to the collector in zipkin.
>
> I am going to make some architecture slides for my talk at ApacheCon
> US, I will post those as soon as they exist (I'll also do a short
> writeup on setting up)
>
> Thanks for checking out htrace!
>
> Colin
>
> On Sat, Feb 28, 2015 at 8:04 PM, Michael Wall <mjwall@gmail.com> wrote:
> > Thanks for the replies.  Not sure if this is useful or not, but here are
> > some notes on how I got htraced running.  I missed the
> > htrace-core/src/go/BUILDING.txt file before, which mentioned these items
> > already.
> >
> > - Install Go
> > Download 1.3.1 from https://golang.org/dl/
> > Unzip somewhere, like /local/opt/go
> > export GOROOT=/local/opt/go
> > export GOPATH="${GOROOT}/packages"
> > add $GOROOT/bin and $GOPATH/bin to $PATH
> > Seems like GOROOT may no longer be needed for newer versions of Go
> >
> > - Install godep
> > Run
> > go get github.com/tools/godep
> > Puts it in GOPATH
> >
> > - Install leveldb
> > Download 1.13.1 from
> >
> https://code.google.com/p/leveldb/downloads/detail?name=leveldb-1.13.0.tar.gz
> > extract, cd into leveldb directory and run make
> > as root from leveldb directory setup lib and include for compiler
> > -- Linux CentOS 6.5
> > cp --preserve=links libleveldb.* /usr/lib64
> > cp -r include/leveldb /usr/include/
> > ldconfig
> > -- Mac 10.10
> > xcode-select --install # to make sure /usr/local/include is included
> > cp -a libleveldb.* /usr/local/lib
> > cp -r include/leveldb /usr/local/include/
> >
> > - Install maven
> > Already had 3.0.5 installed
> >
> > - Get HTrace
> > git clone http://git-wip-us.apache.org/repos/asf/incubator-htrace.git
> > mvn clean package
> >
> > I looked the code a little and still have some questions.
> >
> > 1)  Is REST the only interface for getting span info in and out of
> htraced?
> > 2)  How is HTracedRESTReciever in htrace-htraced/ used or how will it be
> > used?  I didn't see it anywhere in the Java clients.
> > 3)  Is it expected that there will be one htraced process running.  I am
> > thinking of it like the collector in Zipkin,
> > http://twitter.github.io/zipkin/Architecture.html.  Is that correct?  Is
> > there a similar architecture diagram for HTrace to help me see the big
> > picture?  Happy to help make one once I figure it out.
> >
> > Thanks in advance for the help.  Once I understand more, I'll look at the
> > open Jira's if that is the best way to get involved.
> >
> > Mike
> >
> >
> > On Thu, Feb 19, 2015 at 9:40 PM, Colin McCabe <cmccabe@alumni.cmu.edu>
> > wrote:
> >
> >> Hi Michael,
> >>
> >> It's great that you're looking at the project!
> >>
> >> The build requirements are in BUILDING.txt, which says "Requires go
> >> version 1.3.1 or 1.4."  In practice, other versions of go will
> >> probably work, we just haven't tested them.  If you are using any
> >> modern version of Linux, you should be able to use "apt-get install
> >> golang" or "zypper install golang", etc. etc.  Please do let us know
> >> if there are any difficulties building, though... we want this to be
> >> very accessible.
> >>
> >> As Nick commented, you don't need golang to build the Java client
> >> library.  You might find it helpful to use the "LocalFileSpanReceiver"
> >> which just writes trace spans to a local file.  The go code is used in
> >> htraced, which is a daemon which aggregates the spans all in one
> >> place.
> >>
> >> I'm doing some testing now myself with HTrace+Hadoop+HBase and I'll
> >> try to send out my notes about how I set things up.  It will probably
> >> take a week to do that though since there is other stuff on my plate.
> >> But stay tuned.
> >>
> >> re: compiling without golang.  We are planning on moving the htraced
> >> stuff into the htace-htraced subproject rather than htrace-core, where
> >> it is now.  Once that move happens you should be able to just compile
> >> the maven subproject you want.  I'm not sure if we need a maven
> >> profile or not since basically the profile would be equivalent to just
> >> not compiling that subproject.
> >>
> >> best,
> >> Colin
> >>
> >> On Thu, Feb 19, 2015 at 12:26 PM, Michael Wall <mjwall@gmail.com>
> wrote:
> >> > Hi,
> >> >
> >> > I am interested in learning more about HTrace and helping.  Hopefully
> >> this
> >> > is the correct list, I didn't see a users alias.
> >> >
> >> > Specifically, I use Accumulo everyday and am investigating how HTrace
> can
> >> > supplement or replace Accumulo's built-in tracing.
> >> >
> >> > I just checked out the code and tried to build it.  To my surprise, I
> >> need
> >> > to install Golang.  What version should I be using?  Is there
> somewhere I
> >> > can read about why this was chosen and which modules/functionality Go
> >> will
> >> > be used for?
> >> >
> >> > Thanks
> >> >
> >> > Mike Wall
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message