hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Thieme <athi...@athieme.com>
Subject Re: Java submit job to remote server
Date Tue, 12 Feb 2013 22:21:25 GMT
Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included
it below. There certainly appears to be a server running on localhost:9001. At least, I was
able to telnet to that address. While in development, I'm treating the server on localhost
as the remote server. Moving to production, there'd obviously be a different remote server
address configured.

Root Exception stack trace:
java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:375)
	at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
    + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************

On Feb 12, 2013, at 4:22 PM, Nitin Pawar <nitinpawar432@gmail.com> wrote:

> conf.set("mapred.job.tracker", "localhost:9001");
> 
> this means that your jobtracker is on port 9001 on localhost 
> 
> if you change it to the remote host and thats the port its running on then it should
work as expected 
> 
> whats the exception you are getting? 
> 
> 
> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <athieme@athieme.com> wrote:
> I apologize for asking what seems to be such a basic question, but I would use some help
with submitting a job to a remote server.
> 
> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written
some Java code to submit a job. 
> 
> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
> 
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the
exception included below.
> 
> If that line is disabled, then the job is completed. However, in reviewing the hadoop
server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as
processed by the server. Instead, I wonder if my Java code is simply running the necessary
mapper Java code, bypassing the locally installed server.
> 
> Thanks in advance.
> 
> Alex
> 
> public class OfflineDataTool extends Configured implements Tool {
> 
>     public int run(final String[] args) throws Exception {
>         final Configuration conf = getConf();
>         //conf.set("mapred.job.tracker", "localhost:9001");
> 
>         final Job job = new Job(conf);
>         job.setJarByClass(getClass());
>         job.setJobName(getClass().getName());
> 
>         job.setMapperClass(OfflineDataMapper.class);
> 
>         job.setInputFormatClass(TextInputFormat.class);
> 
>         job.setMapOutputKeyClass(Text.class);
>         job.setMapOutputValueClass(Text.class);
> 
>         job.setOutputKeyClass(Text.class);
>         job.setOutputValueClass(Text.class);
> 
>         FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
> 
>         final org.apache.hadoop.fs.Path output = new org.apache.hadoop.fs.Path(args[1]);
>         FileSystem.get(conf).delete(output, true);
>         FileOutputFormat.setOutputPath(job, output);
> 
>         return job.waitForCompletion(true) ? 0 : 1;
>     }
> 
>     public static void main(final String[] args) {
>         try {
>             int result = ToolRunner.run(new Configuration(), new OfflineDataTool(), new
String[]{"offline/input", "offline/output"});
>             log.error("result = {}", result);
>         } catch (final Exception e) {
>             throw new RuntimeException(e);
>         }
>     }
> }
> 
> public class OfflineDataMapper extends Mapper<LongWritable, Text, Text, Text> {
> 
>     public OfflineDataMapper() {
>         super();
>     }
> 
>     @Override
>     protected void map(final LongWritable key, final Text value, final Context context)
throws IOException, InterruptedException {
>         final String inputString = value.toString();
>         OfflineDataMapper.log.error("inputString = {}", inputString);
>     }
> }
> 
> 
> 
> 
> -- 
> Nitin Pawar


Mime
View raw message