hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix.徐 <ygnhz...@gmail.com>
Subject How to run or submit MapReduce Job to Hadoop in my own program?
Date Tue, 17 May 2011 10:47:13 GMT
Hi,all..How can I run a MR job though my own program instead of using
console to submit a job to a real Hadoop env?
I write code like this, this program works fine but i don't think it ran in
my Hadoop env,since nothing was produced in hadoop logs folder.
public int run(String[] args) throws Exception {

                Configuration config = new Configuration();
                config.set("fs.default.name", "hdfs://localhost:54310");
                config.set("mapred.job.tracker","localhost:54311");
                JobConf conf = new JobConf(config,CustomMR.class);
                conf.setJobName("custom aggregation");
                conf.setMapOutputKeyClass(Text.class);
                conf.setMapOutputValueClass(IntWritable.class);

            conf.setOutputFormat(HiveIgnoreKeyTextOutputFormat.class);

            conf.setMapperClass(MapClass.class);

            conf.setReducerClass(Reduce.class);
            TextInputFormat.setInputPaths(conf,args[0]);
            HiveIgnoreKeyTextOutputFormat.setOutputPath(conf, new
Path(args[1]));

            JobClient.runJob(conf);
                //conf
                return 0;
        }
public static void main(String[] args) throws Exception {
            args = new String[]{"/user/hive/warehouse/ptest","/testout"};
            int res = ToolRunner.run(new Configuration(), new CustomMR(),
args);
            System.exit(res);
}
Could some one give me an example?Thanks!

Mime
View raw message