hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Kent <m...@persai.com>
Subject Re: JobClient question
Date Fri, 11 Jul 2008 16:33:25 GMT
The best way to configure all that stuff is in hadoop-site.xml, which
lives in the hadoop conf directory. Make sure that directory is on the
classpath of your application.

On Fri, 2008-07-11 at 11:55 -0400, Larry Compton wrote:
> Thanks. Is this the correct syntax?
> 
> conf.set("mapred.job.tracker", "localhost:54311");
> 
> It does appear to be communicating with the JobTracker now, but I get the
> following stack trace. Is there anything else that needs to be done to
> configure the job?
> 
> Exception in thread "main" org.apache.hadoop.ipc.RemoteException:
> java.io.IOException:
> /home/larry/pkg/hadoop/hdfs/mapred/system/job_200807111114_0001/job.xml: No
> such file or directory
>     at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
>     at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149)
>     at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155)
>     at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136)
>     at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:175)
>     at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:585)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> 
>     at org.apache.hadoop.ipc.Client.call(Client.java:557)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>     at $Proxy0.submitJob(Unknown Source)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:585)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>     at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>     at $Proxy0.submitJob(Unknown Source)
>     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:758)
>     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
>     at jobclient.MaterializeMain.main(MaterializeMain.java:44)
> 
> 
> On Fri, Jul 11, 2008 at 11:41 AM, Shengkai Zhu <geniusjash@gmail.com> wrote:
> 
> > You should provide JobTracker address and port through configuration.
> >
> >
> > On 7/11/08, Larry Compton <lawrence.compton@gmail.com> wrote:
> > >
> > > I'm coming up to speed on the Hadoop APIs. I need to be able to invoke a
> > > job
> > > from within a Java application (as opposed to running from the
> > command-line
> > > "hadoop" executable). The JobConf and JobClient appear to support this
> > and
> > > I've written a test program to configure and run a job. However, the job
> > > doesn't appear to be submitted to the JobTracker. Here's a code excerpt
> > > from
> > > my client...
> > >
> > >        String rdfInputPath = args[0];
> > >        String outputPath = args[1];
> > >        String uriInputPath = args[2];
> > >        String jarPath = args[3];
> > >
> > >        JobConf conf = new JobConf(MaterializeMap.class);
> > >        conf.setJobName("materialize");
> > >
> > >        conf.setOutputKeyClass(Text.class);
> > >        conf.setOutputValueClass(Text.class);
> > >
> > >        conf.setMapperClass(MaterializeMapper.class);
> > >        conf.setCombinerClass(MaterializeReducer.class);
> > >        conf.setReducerClass(MaterializeReducer.class);
> > >        conf.setJar(jarPath);
> > >
> > >        DistributedCache.addCacheFile(new Path(uriInputPath).toUri(),
> > conf);
> > >
> > >        FileInputFormat.setInputPaths(conf, new Path(rdfInputPath));
> > >        FileOutputFormat.setOutputPath(conf, new Path(outputPath));
> > >
> > >        conf.setInputFormat(TextInputFormat.class);
> > >        conf.setOutputFormat(TextOutputFormat.class);
> > >
> > >        JobClient.runJob(conf);
> > >
> > > It seems like I should be providing a URL to the JobTracker somewhere,
> > but
> > > I
> > > can't figure out where to provide the information.
> > >
> > > --
> > > Larry Compton
> > >
> >
> >
> >
> > --
> >
> > 朱盛凯
> >
> > Jash Zhu
> >
> > 复旦大学软件学院
> >
> > Software School, Fudan University
> >
> 
> 
> 


Mime
View raw message