hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "vitthal (Suhas) Gogate (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-3202) Integrating Hadoop Vaidya with JobHistory Server
Date Tue, 18 Oct 2011 18:23:10 GMT
Integrating Hadoop Vaidya with JobHistory Server
------------------------------------------------

                 Key: MAPREDUCE-3202
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202
             Project: Hadoop Map/Reduce
          Issue Type: New Feature
          Components: jobhistoryserver
    Affects Versions: 0.20.205.1
            Reporter: vitthal (Suhas) Gogate


At present jobdetailshistory page served by JobHistory Server provides elementary job analysis
through link "Analyze This job". Hadoop Vaidya provides a detailed analysis of the M/R job
in terms of various execution inefficiencies and the associated remedies that user can easily
understand and fix. Integrating Hadoop Vaidya with JobHistory server would really improve
the usability of this tool and also benefit many naive users understanding various performance
problems and/or best practices violations associated with their job.

Integration would also aim at providing users a convenient interface where they can manage
the existing rules as well as write their own new rules. 

During my tenure at Yahoo, Vaidya tool was successfully deployed in production analyzing tens
of thousands of jobs every day with lot more useful rules than the sample ones present in
the contrib project. Many of these rules are open sourced already (big thanks to Yahoo! MAPREDUCE-1530)
but yet to integrate with the tool.

I will add more design details for this feature in near future as work towards getting prototype
running.. Any thoughts/comments are welcome. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message