htrace-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@apache.org>
Subject Re: Trace through MapReduce?
Date Sat, 26 Nov 2016 19:21:04 GMT
P.S. If you're interested in contributing, adding this MapReduce ID to
spans would be a cool project.  Also, converting accumulo to the latest
version of HTrace (4.0) would be great.

Colin


On Sat, Nov 26, 2016, at 11:20, Colin McCabe wrote:
> Hi Dylan,
> 
> Thanks for trying out HTrace!  We haven't added HTrace support to
> MapReduce yet.  Since MapReduce involves very long-running jobs, there
> is some discussion about the best way to add HTrace support to it. It
> doesn't really fit into the "one trace per request" model that HDFS
> uses.  One promising proposal is to add a tag to all spans that are
> created during a given mapreduce job, that contains an ID which can be
> traced back to the MR job.
> 
> best,
> Colin
> 
> 
> On Sat, Nov 26, 2016, at 05:17, Dylan Hutchison wrote:
> > Hi folks,
> > 
> > I am using HTrace 3 with Accumulo.  I would like to trace through a
> > MapReduce program that uses Accumulo Input/Output formats.  Has anyone
> > done
> > this?  I am using Hadoop 2.7.2, HTrace 3.1.0, Accumulo 1.8.0.
> > 
> > I confirm HTrace 3 is working with client java programs that scan
> > Accumulo.
> > 
> > 
> > I am not sure if Hadoop tracing is working. I added the ZooTraceClient
> > configuration to Hadoop and added the relevant Accumulo jars to Hadoop's
> > classpath, but I don't know if it worked.  (I see a new trace entry
> > called
> > ClientNamenodeProtocol that I never saw before, but it's not proof that
> > Hadoop tracing is working.)
> > 
> > I don't think the trace is being wrapped around the MapReduce mechanisms
> > that exec Mappers and Reducers over Yarn.
> > 
> > Maybe I can make it work by detaching the trace?  Would HTrace work if I
> > detach a trace from one process, record the trace ID, send the trace ID
> > to
> > the mappers and reducers, and then re-attach at the mapper and reducer
> > processes?
> > 
> > Cheers, Dylan

Mime
View raw message