hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naganarasimha Garla <naganarasimha...@gmail.com>
Subject Re: Invoking a MapRed job via Runtime class
Date Fri, 09 Oct 2015 03:20:30 GMT
Hi Ista,
     Well seems more like not a Hadoop related query,
Possible solutions what i can think of is
1. Code refactoring such that some methods are exposed in your MR driver
class and call the method directly.
2. Not sure whats the error in your code but possible reason would be
process's error and out put needs to be read  using Stream gobbler.

P.S. you can have "*job.waitForCompletion(true);" *itself

+ Naga

On Fri, Oct 9, 2015 at 12:36 AM, Istabrak Abdul-Fatah <ifatah@gmail.com>
wrote:

> Hi Naga,
>
> I have tried having the client driver as the mapreduce job itself and as
> you know it works with no issues.
> The intention is to have a java driver that can trigger these mapreduce
> jobs.
> This driver may run inside an app server or something similar (on the
> Hadoop cluster admin node) and respond to remote client calls to trigger
> such jobs since these remote clients are not supposed to have any Hadoop
> jars /config files.
>
> So my question is: Would it be possible to invoke such a mapreduce job
> from a java driver that has no Mapreduce code other than the invocation
> statement?
>
> BR
>
> Ista
>
> On Thu, Oct 8, 2015 at 1:19 PM, Naganarasimha Garla <
> naganarasimha.gr@gmail.com> wrote:
>
>> Hi Ista,
>>      What i meant was to have your "mapReduce" job's run code in your "javaDriver2"'s
>> main method if your intention is to just run the MR job and not wait for
>> the result. May be it would be good(helpful) for you to go through the java
>> docs for Job.
>> Whats the intention of running this  job ? Testing of the MR or the
>> client driver ?
>>
>> + Naga
>>
>> On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <ifatah@gmail.com>
>> wrote:
>>
>>> Hi Naga,
>>> Thx for the reply.
>>> Here is a high level description of what I am trying to achieve (if
>>> possible):
>>>
>>> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
>>> being implemented, compiled and the respective generated jars are there as
>>> well.
>>>
>>> In the same node, I have a java application running and am trying to
>>> start one of the existing mapreduce jobs (as you called it "to sbumit the
>>> job" ).
>>>
>>> The code snippets are provided below.
>>> Currently, it is not working and I am wondering if I have missed
>>> something or have incorrect impl.
>>>
>>> How would the code changes if the calling application would like to wait
>>> for the mapreduce job to complete?
>>>
>>> Thx
>>>
>>> Ista
>>>
>>>
>>> #################################################
>>>
>>> The java application code snippet
>>> ===========================
>>>
>>> public class javaDriver2 {
>>>
>>> public static void main(String[] args) {
>>> Runtime rt = Runtime.getRuntime();
>>> try {
>>> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
>>> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
>>> } catch (IOException e) {
>>> e.printStackTrace();
>>> System.out.println("caught exception" + e.toString());
>>> }
>>> catch (SecurityException se) {
>>> se.printStackTrace();
>>> System.out.println("caught exception" + se.toString());
>>> }
>>>
>>> }
>>>
>>> ///////////////////////////////////
>>>
>>> The Mapreduce job code:
>>> =====================
>>>
>>> public static void main(String[] args) throws Exception {
>>> Configuration conf = new Configuration();
>>> int res = ToolRunner.run(conf, new AvgEight(), args);
>>> System.exit(res);
>>> }
>>>
>>> @Override
>>> public int run(String[] args) throws Exception {
>>>
>>> // When implementing tool
>>> Configuration conf = this.getConf();
>>>
>>> // Create job
>>> Job job = Job.getInstance(conf, "AvgEight");
>>> job.setJarByClass(AvgEight.class);
>>>
>>> // Setup MapReduce job
>>> // Do not specify the number of Reducers
>>> job.setMapperClass(Map2.class);
>>> job.setReducerClass(Reduce2.class);
>>>
>>> job.setOutputKeyClass(Text.class);
>>>
>>> //job.setOutputValueClass(IntWritable.class);
>>> //job.setOutputValueClass(ArrayWritable.class);
>>> job.setOutputValueClass(LongArrayWritable.class);
>>>
>>> job.setInputFormatClass(TextInputFormat.class);
>>> job.setOutputFormatClass(TextOutputFormat.class);
>>>
>>> FileInputFormat.addInputPath(job, new Path(args[0]));
>>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>>
>>> job.submit();
>>> return 1;
>>> }
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
>>> garlanaganarasimha@huawei.com> wrote:
>>>
>>>> Hi Ista,
>>>> IIUC you just want it to be submitted and not wait till it finish and
>>>> print the stats ?
>>>> Basically sample code which you might be referring will be creating the
>>>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>>>> configurations calling
>>>> *job.waitForCompletion(true);* Instead you can just call
>>>> *job.submit();*
>>>>
>>>> Regards,
>>>> Naga
>>>> ------------------------------
>>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>>> *Sent:* Wednesday, October 07, 2015 19:58
>>>> *To:* user@hadoop.apache.org
>>>> *Subject:* Invoking a MapRed job via Runtime class
>>>>
>>>> Greetings to all,
>>>> Is it possible to invoke a MapReduce job from another java class using
>>>> the runtime command? If not, what are the alternatives?
>>>>
>>>> Here is a sample code snippet:
>>>>
>>>> public static void main(String[] args) {
>>>>      Runtime rt = Runtime.getRuntime();
>>>> try {
>>>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>>>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>>>> } catch (IOException e) {
>>>> e.printStackTrace();
>>>> }
>>>>
>>>> Regards,
>>>>
>>>> Ista
>>>>
>>>
>>>
>>
>

Mime
View raw message