hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuti Awasthi <stutiawas...@hcl.com>
Subject RE: Calling one MR job within another MR job
Date Wed, 04 Apr 2012 12:01:36 GMT
Hi Ashwanth,

My scenario is not resolved by chaining jobs as in chaining : Output of one MR job is input
in other MR job.
Neither I can use JobControl Api as this tells Job1 to wait till Job2 is complete.
In my scenario processing of each line File1 is dependent on simultaneous processing of File2.

From: ashwanth.kumar@gmail.com [mailto:ashwanth.kumar@gmail.com] On Behalf Of Ashwanth Kumar
Sent: Wednesday, April 04, 2012 4:34 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Calling one MR job within another MR job

Have you tired using JobConf / JobClient for starting new jobs?

Also refer here -  http://developer.yahoo.com/hadoop/tutorial/module4.html#chaining  on Job
Chaining.
On Wed, Apr 4, 2012 at 4:19 PM, Stuti Awasthi <stutiawasthi@hcl.com<mailto:stutiawasthi@hcl.com>>
wrote:
Hi Ashwanth,

No I have not tried oozie. I want to attain this simply through Java Map Reduce jobs.
Any ideas?

From: ashwanth.kumar@gmail.com<mailto:ashwanth.kumar@gmail.com> [mailto:ashwanth.kumar@gmail.com<mailto:ashwanth.kumar@gmail.com>]
On Behalf Of Ashwanth Kumar
Sent: Wednesday, April 04, 2012 4:13 PM
To: mapreduce-user@hadoop.apache.org<mailto:mapreduce-user@hadoop.apache.org>
Subject: Re: Calling one MR job within another MR job

Have you tired using Oozie < http://incubator.apache.org/oozie/ >?

On Wed, Apr 4, 2012 at 4:04 PM, Stuti Awasthi <stutiawasthi@hcl.com<mailto:stutiawasthi@hcl.com>>
wrote:
Hi all,

We have a usecase in which I start with first MR1 job with input file as File1.txt, and from
this job, call another MR2 job with input as File2.txt
So :
MRjob1{
Map(){
MRJob2(File2.txt)
}
}

MRJob2{
Processing....
}

My queries are is this kind of approach is possible and how much are the implications from
the performance perspective.


Regards,
Stuti Awasthi
HCL Comnet Systems and Services Ltd
F-8/9 Basement, Sec-3,Noida.


________________________________
::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named
recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or
opinions presented in
this email are solely those of the author and may not necessarily reflect the opinions of
HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and
/ or publication of
this message without the prior written consent of the author of this e-mail is strictly prohibited.
If you have
received this email in error please delete it and notify the sender immediately. Before opening
any mail and
attachments please check them for viruses and defect.

-----------------------------------------------------------------------------------------------------------------------



--

Ashwanth Kumar / ashwanthkumar.in<http://ashwanthkumar.in/>





--

Ashwanth Kumar / ashwanthkumar.in<http://ashwanthkumar.in/>



Mime
View raw message