hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jie Li (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (MAPREDUCE-4365) Shipping Profiler Libraries by DistributedCache
Date Thu, 28 Jun 2012 01:12:48 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jie Li resolved MAPREDUCE-4365.

          Resolution: Fixed
    Target Version/s:   (was: 1.1.0)

One way is to include the profiler library into the job jar and use relative path like "../../foo.library"
to locate it.

Thanks Deveraj, Sid, Vinod and everyone!
> Shipping Profiler Libraries by DistributedCache
> -----------------------------------------------
>                 Key: MAPREDUCE-4365
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4365
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>    Affects Versions: 1.0.3
>            Reporter: Jie Li
> Hadoop profiling is great for performance tuning and debugging, but currently we can
only use Java built-in profilers such as HProf, and for other profilers we need to install
them on all slave nodes first, which is inconvenient for large clusters and sometimes impossible
for production clusters. 
> Supporting shipping profiler libraries using DistributedCache will solve this problem.
For example, in mapred.task.profile.params, we specify a profiler library from the DistributedCache
using special place holders such as <foo.jar>, and Hadoop can look at the DistributedCache
to replace <foo.jar> with the localized path before launching the child jvm.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message