hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Schlosser" <swschlos...@gmail.com>
Subject Re: Read counters in main?
Date Wed, 03 Oct 2007 17:24:10 GMT
Thanks for the help.  This seems to do what I want, but now I'm having
another problem.  I get a null pointer exception in JobClient.getFs(),
which is called by submitJob().

Is there something else I need to set in the jobConf object I pass to
submitJob()?  It works fine when I use JobClient.runJob() instead of
submitJob()...

Here is the stack from when the exception is thrown:

java.lang.NullPointerException
        at org.apache.hadoop.mapred.JobClient.getFs(JobClient.java:236)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:272)
        at net.intelresearch.cvmHadoop.KeyedByLocationalCode$ReduceOnlyDriver.main(Unknown
Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
        at net.intelresearch.cvmHadoop.Usage.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:155)


On 10/3/07, Arun C Murthy <arunc@yahoo-inc.com> wrote:
> On Wed, Oct 03, 2007 at 10:20:32AM -0400, Steve Schlosser wrote:
> >Hello all
> >
> >What is the procedure for reading counters after a job has run?  I
> >want to be able to run my job using JobClient.runJob(), then read out
> >some counters to make a decision about whether to run another
> >reduction or exit.  I can think of some hacky ways to do it, but there
> >must be some correct way to read the counters.
> >
>
> Take a look at: http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/JobClient.html#submitJob(org.apache.hadoop.mapred.JobConf).
>
> That returns a RunningJob, then use http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/RunningJob.html#getCounters()
>
> Arun
>
> >Thanks!
> >
> >-steve
>

Mime
View raw message