hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Jurgens <david.jurg...@gmail.com>
Subject Re: Profiling Hadoop Map Reduce with the 20.2 API
Date Mon, 16 Aug 2010 20:34:37 GMT
Hi Hemanth,

  It looks like calling configuration.setBoolean("mapred.task.profile",
true) will enable profiling with the 20.2 APIs.  I am able to see the
profiling output when I check the web interface.  Thanks for your help!  Is
there a good place to document this setting so others can find this
information?

  Thanks,
  David


On Thu, Aug 12, 2010 at 10:16 PM, Hemanth Yamijala <yhemanth@gmail.com>wrote:

>  Hi,
>
> >   I recently started developing with Hadoop using the 20.2 API.  I'm
> looking
> > to profile one of my jobs but I haven't been able to find any
> documentation
> > about how to do this.  For the earlier (deprecated) API, there's some
> > documentation on how to profile with the JobConf class
> > (
> http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Profiling
> ).
> > Is there anything equivalent in 20.2, or is there another process to use
> for
> > it?  Search engines and javadoc searching haven't turned up anything so
> far
> > for me.
>
> All APIs in the deprecated JobConf class are wrappers around the
> Configuration API that you probably are using in the 20.2 new API. For
> e.g. :
>
> setProfileEnabled = configuration.setBoolean("mapred.task.profile", value)
> setProfileTaskRange = configuration.set("mapred.task.profile.maps",
> value) or configuration.set("mapred.task.profile.reduces", value)
> setProfileParams = configuration.set("mapred.task.profile.params", value)
>
> Could you try setting these parameters in the Configuration you might
> be using for the Job and see if that works ?
>
> Thanks
> Hemanth
>

Mime
View raw message