hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Libo Yu <yu_l...@hotmail.com>
Subject RE: how to customize hadoop configuration for a job?
Date Wed, 02 Apr 2014 12:53:48 GMT
That is exactly what I did. But the command line parameters are ignored.
The hadoop I am using is 2.0.0-cdh4.1.2. The property I tried is io.sort.mb.


From: unmeshabiju@gmail.com
Date: Wed, 2 Apr 2014 08:50:10 +0530
Subject: Re: how to customize hadoop configuration for a job?
To: user@hadoop.apache.org

Hi Libo, You can implement your driver code using ToolRunner.So that you can pass your extra
configuration through command line instead of editing your code all the time.

Driver code

----------------public class WordCount extends Configured implements Tool {

  public static void main(String[] args) throws Exception {            int exitCode = ToolRunner.run(new
Configuration(), new WordCount(), args);

            System.exit(exitCode);  }  public int run(String[] args) throws Exception {

            if (args.length != 2) {                    System.out.printf("Usage: %s [generic
options] <input dir> <output dir>\n", getClass().getSimpleName());

             return -1;           }         Job job = new Job(getConf());

         job.setJarByClass(WordCount.class);         job.setJobName("Word Count");

         FileInputFormat.setInputPaths(job, new Path(args[0]));         FileOutputFormat.setOutputPath(job,
new Path(args[1]));

         job.setMapperClass(WordMapper.class);         job.setReducerClass(SumReducer.class);

         job.setMapOutputKeyClass(Text.class);         job.setMapOutputValueClass(IntWritable.class);

         job.setOutputKeyClass(Text.class);         job.setOutputValueClass(IntWritable.class);

         boolean success = job.waitForCompletion(true);        return success ? 0 : 1;

command line

-------------------$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir myoutputdir

This is a better practise.

Happy Hadooping.

Thanks & Regards

Unmesha Sreeveni U.B
Hadoop, Bigdata DeveloperCenter for Cyber Security | Amrita Vishwa Vidyapeetham


View raw message