Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ABB9F18480 for ; Thu, 8 Oct 2015 14:10:21 +0000 (UTC) Received: (qmail 87479 invoked by uid 500); 8 Oct 2015 14:10:15 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 87350 invoked by uid 500); 8 Oct 2015 14:10:15 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 87337 invoked by uid 99); 8 Oct 2015 14:10:14 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Oct 2015 14:10:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 5F7A2C4BD9 for ; Thu, 8 Oct 2015 14:10:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id ZWBei0NsLY8Y for ; Thu, 8 Oct 2015 14:10:10 +0000 (UTC) Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com [209.85.212.181]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id BF3A02304E for ; Thu, 8 Oct 2015 14:10:09 +0000 (UTC) Received: by wiclk2 with SMTP id lk2so30364772wic.0 for ; Thu, 08 Oct 2015 07:10:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=oj2RRsdt9qcevNUwUzkZ8XeAN1G4Z6dZ13o6iPKnjbY=; b=GwEd43SfemjpjDTJBGaRt70+FT/YsTIU7plys7RfALHwOSldUUxdKTsdZdAjommJ9I szux5MyrGdnxCfOGYDrSDxpExFZ6lg26ar1adSFTxLpNVB5aY+OPhmApZL/doEG24iH4 qibMHOT7WXoc1bbtcEX0RMsOBjHDa+bP/vg/BZ7RKSSf/LZ/pvNlZzUwZevIDM+YGpx2 PAbOTcapznK8BCFMXxEILHYyHW/dZcfQH76Ln+BhINZihazAxHR9MkDdKIPp3fbezRqw xpKIwmCtysiueg1UdSs5iYYTe0zWppgYgXirmXSi3mp48yEtosM7rGn9pSqQrMJiv9L2 9+Cg== MIME-Version: 1.0 X-Received: by 10.180.39.193 with SMTP id r1mr4192355wik.57.1444313408026; Thu, 08 Oct 2015 07:10:08 -0700 (PDT) Received: by 10.27.136.86 with HTTP; Thu, 8 Oct 2015 07:10:07 -0700 (PDT) In-Reply-To: References: Date: Thu, 8 Oct 2015 10:10:07 -0400 Message-ID: Subject: Re: Invoking a MapRed job via Runtime class From: Istabrak Abdul-Fatah To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c2b930a1407d0521986b77 --001a11c2b930a1407d0521986b77 Content-Type: text/plain; charset=UTF-8 Hi Naga, Thx for the reply. Here is a high level description of what I am trying to achieve (if possible): On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs being implemented, compiled and the respective generated jars are there as well. In the same node, I have a java application running and am trying to start one of the existing mapreduce jobs (as you called it "to sbumit the job" ). The code snippets are provided below. Currently, it is not working and I am wondering if I have missed something or have incorrect impl. How would the code changes if the calling application would like to wait for the mapreduce job to complete? Thx Ista ################################################# The java application code snippet =========================== public class javaDriver2 { public static void main(String[] args) { Runtime rt = Runtime.getRuntime(); try { Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar -Dtest=\"9999\" /user/yarn/input/sample.csv output"); } catch (IOException e) { e.printStackTrace(); System.out.println("caught exception" + e.toString()); } catch (SecurityException se) { se.printStackTrace(); System.out.println("caught exception" + se.toString()); } } /////////////////////////////////// The Mapreduce job code: ===================== public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); int res = ToolRunner.run(conf, new AvgEight(), args); System.exit(res); } @Override public int run(String[] args) throws Exception { // When implementing tool Configuration conf = this.getConf(); // Create job Job job = Job.getInstance(conf, "AvgEight"); job.setJarByClass(AvgEight.class); // Setup MapReduce job // Do not specify the number of Reducers job.setMapperClass(Map2.class); job.setReducerClass(Reduce2.class); job.setOutputKeyClass(Text.class); //job.setOutputValueClass(IntWritable.class); //job.setOutputValueClass(ArrayWritable.class); job.setOutputValueClass(LongArrayWritable.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.submit(); return 1; } On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) < garlanaganarasimha@huawei.com> wrote: > Hi Ista, > IIUC you just want it to be submitted and not wait till it finish and > print the stats ? > Basically sample code which you might be referring will be creating the > instance of "org.apache.hadoop.mapreduce.Job" and setting with the req > configurations calling > *job.waitForCompletion(true);* Instead you can just call *job.submit();* > > Regards, > Naga > ------------------------------ > *From:* Istabrak Abdul-Fatah [ifatah@gmail.com] > *Sent:* Wednesday, October 07, 2015 19:58 > *To:* user@hadoop.apache.org > *Subject:* Invoking a MapRed job via Runtime class > > Greetings to all, > Is it possible to invoke a MapReduce job from another java class using the > runtime command? If not, what are the alternatives? > > Here is a sample code snippet: > > public static void main(String[] args) { > Runtime rt = Runtime.getRuntime(); > try { > Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar > -Dtest=\"9999\" /user/yarn/input/samplefile.csv output"); > } catch (IOException e) { > e.printStackTrace(); > } > > Regards, > > Ista > --001a11c2b930a1407d0521986b77 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Naga,
Thx for the reply.
Here is a high l= evel description of what I am trying to achieve (if possible):
On node A, I have the Hadoop/Yarn v2.7 running with the mapred= uce jobs being implemented, compiled and the respective generated jars are = there as well.

In the same node, I have a java app= lication running and am trying to start one of the existing mapreduce jobs = (as you called it "to sbumit the job" ).

The code snippets are provided below.
Currently, it is not worki= ng and I am wondering if I have missed something or have incorrect impl.

How would the code changes if the calling applicatio= n would like to wait for the mapreduce job to complete?

Thx=C2=A0

Ista

=
#################################################
=
The java application code snippet=C2=A0
=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

public class javaDriver2 {

=
public static void = main(String[] args) {
= Runtime rt =3D R= untime.getRuntime();
=
try = {
Process p2 =3D rt.exe= c("yarn jar /opt/yarn/my_examples/AvgEight.jar -Dtest=3D\"9999\&q= uot; /user/yarn/input/sample.csv output");
} catch (IOException e) {
e.printStackTrace();
=
System.out.printl= n("caught exception" + e.toString());
}
catch (SecurityException se) {
se.printStackTrace();
<= span class=3D"" style=3D"white-space:pre"> System.out.println("= ;caught exception" + se.toString());
}

}

//////////= /////////////////////////

The Mapreduce job code:<= /div>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

public static void main(String[] args) throws = Exception {
<= /div>
Configuration conf =3D ne= w Configuration();
<= /span>int res =3D ToolRunner.run(conf, new AvgEight(), args);
System.exit(res);
}

=
@Override
public int run(String[]= args) throws Exception {

// When implementing tool
Configuration conf =3D this.getC= onf();

= // Create job
Job job =3D Job.getInstance(conf, "AvgEight");
job.setJarByClass(Av= gEight.class);

// Set= up MapReduce job
// Do not specify the number of Reducers
job.setMapperClass(Map2.class);
<= span class=3D"" style=3D"white-space:pre"> job.setReducerClass(Redu= ce2.class);=C2=A0

job.setOutputKeyClass(Text.class);

=
//job.setOutputVal= ueClass(IntWritable.class);
//job.setOutputValueClass(ArrayWritable.class);
job.setOutputValueClass(L= ongArrayWritable.class);

job.setInputFormatClass(TextInputFormat.class);<= /div>
job.setOutput= FormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, = new Path(args[0]));
= FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.submit= ();
return 1;=
}



= On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <gar= lanaganarasimha@huawei.com> wrote:
Hi=C2=A0Ista,
IIUC you just want it to be submitted a= nd not wait till it finish and print the stats ?=C2=A0
Basicall= y sample code which you might be referring will be creating the instance of= "org.apache.hadoop.mapreduce.Job" and setting with the req confi= gurations calling=C2=A0
job.w= aitForCompletion(true); Instead you can just call=C2=A0job.submit(= );

R= egards,
N= aga
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Wednesday, October 07, 2015 19:58
To: user= @hadoop.apache.org
Subject: Invoking a MapRed job via Runtime class

Greetings to all,
Is it possible to invoke a MapReduce job from another java class using= the runtime command? If not, what are the alternatives?

Here is a sample code snippet:

public static void main(String[] args) {
=C2=A0 =C2=A0 =C2=A0Runtim= e rt =3D Runtime.getRuntime();
try {
Process p =3D=C2=A0rt.exec= ("yarn jar /opt/yarn/my_examples/AvgSeven.jar -Dtest=3D\"9999\&qu= ot; /user/yarn/input/samplefile.csv output");
} catch (IOException e) {<= /div>
e.printStackTrace();
}

Regards,

Ista

--001a11c2b930a1407d0521986b77--