hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "vitthal (Suhas) Gogate (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
Date Mon, 30 Sep 2013 05:49:36 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

vitthal (Suhas) Gogate updated MAPREDUCE-3202:
----------------------------------------------

    Description: Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various
execution inefficiencies and the associated remedies that user can easily understand and fix.
This Jira patch integrates it with Job History UI under Hadoop 2.0 branch.  (was: At present
jobdetailshistory page served by JobHistory Server provides elementary job analysis through
link "Analyze This job". Hadoop Vaidya provides a detailed analysis of the M/R job in terms
of various execution inefficiencies and the associated remedies that user can easily understand
and fix. Integrating Hadoop Vaidya with JobHistory server would really improve the usability
of this tool and also benefit many naive users understanding various performance problems
and/or best practices violations associated with their job.

Integration would also aim at providing users a convenient interface where they can manage
the existing rules as well as write their own new rules. 

During my tenure at Yahoo, Vaidya tool was successfully deployed in production analyzing tens
of thousands of jobs every day with lot more useful rules than the sample ones present in
the contrib project. Many of these rules are open sourced already (big thanks to Yahoo! MAPREDUCE-1530)
but yet to integrate with the tool.

I will add more design details for this feature in near future as work towards getting prototype
running.. Any thoughts/comments are welcome. )

> Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 
> ------------------------------------------------------------
>
>                 Key: MAPREDUCE-3202
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: jobhistoryserver
>    Affects Versions: 0.20.205.0, 1.0.0
>            Reporter: vitthal (Suhas) Gogate
>
> Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution
inefficiencies and the associated remedies that user can easily understand and fix. This Jira
patch integrates it with Job History UI under Hadoop 2.0 branch.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message