htrace-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@apache.org>
Subject Re: Trace through MapReduce?
Date Sat, 26 Nov 2016 19:20:10 GMT
Hi Dylan,

Thanks for trying out HTrace!  We haven't added HTrace support to
MapReduce yet.  Since MapReduce involves very long-running jobs, there
is some discussion about the best way to add HTrace support to it. It
doesn't really fit into the "one trace per request" model that HDFS
uses.  One promising proposal is to add a tag to all spans that are
created during a given mapreduce job, that contains an ID which can be
traced back to the MR job.

best,
Colin


On Sat, Nov 26, 2016, at 05:17, Dylan Hutchison wrote:
> Hi folks,
> 
> I am using HTrace 3 with Accumulo.  I would like to trace through a
> MapReduce program that uses Accumulo Input/Output formats.  Has anyone
> done
> this?  I am using Hadoop 2.7.2, HTrace 3.1.0, Accumulo 1.8.0.
> 
> I confirm HTrace 3 is working with client java programs that scan
> Accumulo.
> 
> 
> I am not sure if Hadoop tracing is working. I added the ZooTraceClient
> configuration to Hadoop and added the relevant Accumulo jars to Hadoop's
> classpath, but I don't know if it worked.  (I see a new trace entry
> called
> ClientNamenodeProtocol that I never saw before, but it's not proof that
> Hadoop tracing is working.)
> 
> I don't think the trace is being wrapped around the MapReduce mechanisms
> that exec Mappers and Reducers over Yarn.
> 
> Maybe I can make it work by detaching the trace?  Would HTrace work if I
> detach a trace from one process, record the trace ID, send the trace ID
> to
> the mappers and reducers, and then re-attach at the mapper and reducer
> processes?
> 
> Cheers, Dylan

Mime
View raw message