hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devaraj k <devara...@huawei.com>
Subject RE: understanding hadoop job submission
Date Wed, 25 Apr 2012 10:25:50 GMT
You can submit the job using any one of the below ways,

1. If you submit the job using JobClient, you need to create JobConf and submit the job using
JobClient.runJob(JobConf conf) API.

2. Also you can submit the job by creating instance for Job by passing Configuration object
and submit(using submit() or waitForCompletion()) as you mentioned in the below code. This
case no need to create an instance for JobConf.

Thanks
Devaraj

________________________________________
From: Arindam Choudhury [arindamchoudhury0@gmail.com]
Sent: Wednesday, April 25, 2012 3:27 PM
To: common-user@hadoop.apache.org
Subject: Re: understanding hadoop job submission

Hi,

The code is:

public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();
    if (otherArgs.length != 2) {
      System.err.println("Usage: wordcount <in> <out>");
      System.exit(2);
    }
    Job job = new Job(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
    FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }

I understand it now. But, is it possible to write a program using the
JobClient to submit the hadoop job?

To do that I have to create a JobConf manually. Am I thinking right?

Arindam

On Wed, Apr 25, 2012 at 10:56 AM, Devaraj k <devaraj.k@huawei.com> wrote:

> Hi Arindam,
>
>    hadoop jar jarFileName MainClassName
>
> The above command will not submit the job. This command only executes the
> jar file using the Main Class(Main-class present in manifest info if
> available otherwise class name(i.e MainClassName in the above command)
> passed as an argument. If we give any additional arguments in the command,
> those will be passed to the Main class args.
>
>   We can have a job submission code in the Main Class or any of the
> classes in the jar file. You can take a look into WordCount example for job
> submission info.
>
>
> Thanks
> Devaraj
>
> ________________________________________
> From: Arindam Choudhury [arindamchoudhury0@gmail.com]
> Sent: Wednesday, April 25, 2012 2:14 PM
> To: common-user
> Subject: understanding hadoop job submission
>
> Hi,
>
> I am new to hadoop and I am trying to understand hadoop job submission.
>
> We submit the job using:
>
> hadoop jar some.jar name input output
>
> this in turn invoke the RunJar . But in RunJar I can not find any
> JobSubmit() or any call to JobClient.
>
> Then, how the job gets submitted to the JobTracker?
>
> -Arindam
>

Mime
View raw message