hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Balog <doug.hdp...@dugos.com>
Subject Re: Listing Hadoop Job History Statistics
Date Thu, 12 Aug 2010 04:23:21 GMT
I don't know if this is the best way, but this is how I do it.

Configuration  conf = new Configuration();
JobClient jobClient = new JobClient(new InetSocketAddress("jobTracker",9001),conf);
jobClient.setConf(conf); // Bug in constructor, doesn't set conf.

 for(JobStatus js: jobClient.getAllJobs()){
    // We only care about completed jobs.
                if(!js.isJobComplete()){
                    continue;
                } 
                // Do stuff on jobStatus.
               :
	       :
 }

You can also scrape info from http://jobtracker:50030/jobhistory.jsp

Or read it from the job's outputDir/_log/ directory.

Cheers,

Doug


On Aug 11, 2010, at 11:54 AM, Scott Whitecross wrote:

> Hi -
> 
> What's the best way to list and query information on Hadoop job histories?
> For example, I'd like to see the job names from the past week against a
> Hadoop cluster I'm using.   I don't see an API call or a way through the
> command line to pull the information.  Is the best way writing a quick
> script to process the job history files?
> 
> Thanks.
> Scott


Mime
View raw message