hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Balog <doug.hdp...@dugos.com>
Subject Re: Listing Hadoop Job History Statistics
Date Thu, 12 Aug 2010 04:23:21 GMT
I don't know if this is the best way, but this is how I do it.

Configuration  conf = new Configuration();
JobClient jobClient = new JobClient(new InetSocketAddress("jobTracker",9001),conf);
jobClient.setConf(conf); // Bug in constructor, doesn't set conf.

 for(JobStatus js: jobClient.getAllJobs()){
    // We only care about completed jobs.
                // Do stuff on jobStatus.

You can also scrape info from http://jobtracker:50030/jobhistory.jsp

Or read it from the job's outputDir/_log/ directory.



On Aug 11, 2010, at 11:54 AM, Scott Whitecross wrote:

> Hi -
> What's the best way to list and query information on Hadoop job histories?
> For example, I'd like to see the job names from the past week against a
> Hadoop cluster I'm using.   I don't see an API call or a way through the
> command line to pull the information.  Is the best way writing a quick
> script to process the job history files?
> Thanks.
> Scott

View raw message