hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bart Vandewoestyne <Bart.Vandewoest...@telenet.be>
Subject Re: getting counters from specific hadoop jobs
Date Thu, 23 Oct 2014 14:04:32 GMT
On 10/23/2014 02:56 PM, Dieter De Witte wrote:
> Maybe you could use job -list or job -history to get a list of the
> jobids and extract it from there?

That was indeed one of the methods I was thinking of, but I cannot think 
of a reliable way of implementing it.

Suppose I start a job with hadoop jar, and I wait until it is finished 
and then use `mapred job -list all` to somehow find out the job-id of my 
job that just finished.  Then how do I know what line in the output of 
`mapred job -list all` corresponds to the job I executed?  Even if the 
job output list would be sorted by start time, then I cannot be sure 
that the last started job is mine because another user could have 
started another job after me...

A mechanism that would easily allow a user to get the job-id from a job 
that he just started, would be nice to have.  Doesn't this exist?

Maybe grepping through the output of `mapred job -history all` would be 
the best solution to get to the counter information?  Unfortunately, I 
currently cannot test this approach as I am experiencing the following 

bart@sandy-quad-1:~$ mapred job -history all 
14/10/23 16:03:12 INFO client.RMProxy: Connecting to ResourceManager at 
Ignore unrecognized file: 0050GB
Exception in thread "main" java.io.IOException: Unable to initialize 
History Viewer
	at org.apache.hadoop.mapreduce.tools.CLI.viewHistory(CLI.java:470)
	at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:313)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1239)
Caused by: java.io.IOException: Unable to initialize History Viewer
	... 5 more


Kind regards,

View raw message