hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Thieme <athi...@athieme.com>
Subject Java submit job to remote server
Date Tue, 12 Feb 2013 21:11:01 GMT
I apologize for asking what seems to be such a basic question, but I would use some help with
submitting a job to a remote server.

I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written
some Java code to submit a job. 

Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.

If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception
included below.

If that line is disabled, then the job is completed. However, in reviewing the hadoop server
administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed
by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java
code, bypassing the locally installed server.

Thanks in advance.

Alex

public class OfflineDataTool extends Configured implements Tool {

    public int run(final String[] args) throws Exception {
        final Configuration conf = getConf();
        //conf.set("mapred.job.tracker", "localhost:9001");

        final Job job = new Job(conf);
        job.setJarByClass(getClass());
        job.setJobName(getClass().getName());

        job.setMapperClass(OfflineDataMapper.class);

        job.setInputFormatClass(TextInputFormat.class);

        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(Text.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));

        final org.apache.hadoop.fs.Path output = new org.apache.hadoop.fs.Path(args[1]);
        FileSystem.get(conf).delete(output, true);
        FileOutputFormat.setOutputPath(job, output);

        return job.waitForCompletion(true) ? 0 : 1;
    }

    public static void main(final String[] args) {
        try {
            int result = ToolRunner.run(new Configuration(), new OfflineDataTool(), new String[]{"offline/input",
"offline/output"});
            log.error("result = {}", result);
        } catch (final Exception e) {
            throw new RuntimeException(e);
        }
    }
}

public class OfflineDataMapper extends Mapper<LongWritable, Text, Text, Text> {

    public OfflineDataMapper() {
        super();
    }

    @Override
    protected void map(final LongWritable key, final Text value, final Context context) throws
IOException, InterruptedException {
        final String inputString = value.toString();
        OfflineDataMapper.log.error("inputString = {}", inputString);
    }
}


Mime
View raw message