hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: InvalidJobConfException
Date Fri, 08 Jun 2012 10:50:41 GMT
Hi Huanchen,

Just set your output format class to NullOutputFormat
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/output/NullOutputFormat.html
if you don't need any direct outputs to HDFS/etc. from your M/R
classes.

On Fri, Jun 8, 2012 at 4:16 PM, huanchen.zhang
<huanchen.zhang@ipinyou.com> wrote:
> Hi,
>
> Here I'm developing a MapReduce web crawler which reads url lists and writes html to
MongoDB.
> So, each map read one url list file, get the html and insert to MongoDB. There is no
reduce and no output of map. So, how to set the output directory in this case? If I do not
set the output directory, it gives me following exception,
>
> Exception in thread "main" org.apache.hadoop.mapred.InvalidJobConfException: Output directory
not set.
>        at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:123)
>        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:872)
>        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
>        at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
>        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
>        at com.ipinyou.data.preprocess.mapreduce.ExtractFeatureFromURLJob.main(ExtractFeatureFromURLJob.java:56)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
>
> Thank you !
>
> Best,
> Huanchen
>
>
> 2012-06-08
>
>
>
> huanchen.zhang



-- 
Harsh J

Mime
View raw message