Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A5A7D9A4C for ; Wed, 4 Apr 2012 12:52:00 +0000 (UTC) Received: (qmail 24847 invoked by uid 500); 4 Apr 2012 12:51:59 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 24783 invoked by uid 500); 4 Apr 2012 12:51:59 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 24773 invoked by uid 99); 4 Apr 2012 12:51:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Apr 2012 12:51:59 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of hadoop@gmx.com designates 213.165.64.42 as permitted sender) Received: from [213.165.64.42] (HELO mailout-eu.gmx.com) (213.165.64.42) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 04 Apr 2012 12:51:50 +0000 Received: (qmail 6315 invoked by uid 0); 4 Apr 2012 12:47:09 -0000 Received: from 59.160.60.241 by rms-eu011.v300.gmx.net with HTTP Content-Type: multipart/alternative; boundary="========GMXBoundary27681333543628550753" Date: Wed, 04 Apr 2012 14:47:08 +0200 From: jagatsingh@gmail.com Message-ID: <20120404124708.27680@gmx.com> MIME-Version: 1.0 Subject: RE: Calling one MR job within another MR job To: mapreduce-user@hadoop.apache.org X-Authenticated: #136012697 X-Flags: 0001 X-Mailer: GMX.com Web Mailer x-registered: 0 X-GMX-UID: mycOb7RreSEqIfgAcHwhWF9+IGRvb0Ac --========GMXBoundary27681333543628550753 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Hello Stuti The way you have explained it seems we can think about caching the file2 already in nodes. -- Just out of context , In the same way replicated joins are being handled in Pig in which one file (file2) to be joined is cached in the memory by file1. Regards Jagat ----- Original Message ----- From: Stuti Awasthi Sent: 04/04/12 07:55 AM To: mapreduce-user@hadoop.apache.org Subject: RE: Calling one MR job within another MR job Hi Ravi, There is no job dependency so I cannot use chaining MR or JobControl as you suggested. I have 2 relatively big files, I start processing with File1 as input to MR1 job , now this processing required to find the data from File2. One way to do is loop through File2 and get the data. Other way to pass File2 in MR2 job for parallel processing. Second option is making hinting me to call an MR2 job inside from MR1 job. I am sure this is the common problem that people usually face. What is the best way to resolve this kind of issue. Thanks From: Ravi teja ch n v [mailto:raviteja.chnv@huawei.com] *Sent:* Wednesday, April 04, 2012 4:35 PM *To:* mapreduce-user@hadoop.apache.org *Subject:* RE: Calling one MR job within another MR job Hi Stuti, If you are looking for MRjob2 to run after MRjob1, ie the job dependency, you can use JobControl API, where you can manage the dependencies. Calling another Job from a Mapper is not a good idea. Thanks, Ravi Teja ----------------------------------------------------------------- From: Stuti Awasthi [stutiawasthi@hcl.com] *Sent:* 04 April 2012 16:04:19 *To:* mapreduce-user@hadoop.apache.org *Subject:* Calling one MR job within another MR job Hi all, We have a usecase in which I start with first MR1 job with input file as File1.txt, and from this job, call another MR2 job with input as File2.txt So : MRjob1{ Map(){ MRJob2(File2.txt) } } MRJob2{ Processing…. } My queries are is this kind of approach is possible and how much are the implications from the performance perspective. Regards, Stuti Awasthi HCL Comnet Systems and Services Ltd F-8/9 Basement, Sec-3,Noida. ----------------------------------------------------------------- ::DISCLAIMER:: ----------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any mail and attachments please check them for viruses and defect. ----------------------------------------------------------------------------------------------------------------------- --========GMXBoundary27681333543628550753 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello St= uti
=20
=20 The way you have explained it seems we can think about caching the file2 al= ready in nodes.
=20
=20 -- Just out of context , In the same way replicated joins are being handled= in Pig in which one file (file2) to be joined is cached in the memory by f= ile1.
=20
=20 Regards
=20
=20 Jagat
=20
=20

=20 =C2=A0

=20
=20

=20 ----- = Original Message -----

=20

=20 From: = Stuti Awasthi

=20

=20 Sent: = 04/04/12 07:55 AM

=20

=20 To: ma= preduce-user@hadoop.apache.org

=20

=20 Subjec= t: RE: Calling one MR job within another MR job

=20
=20
=20
=20

=20 Hi Ravi,

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 There is no job dependency so I cannot us= e chaining MR or JobControl as you suggested.

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 I have 2 relatively big files, I start pr= ocessing with File1 as input to MR1 job , now this processing required to f= ind the data from File2. One way to do is loop through File2 and get the da= ta. Other way to pass File2 in MR2 job for parallel processing.

= =20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 Second option is making hinting me to cal= l an MR2 job inside from MR1 job. I am sure this is the common problem that= people usually face. What is the best way to resolve this  kind of is= sue.

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 Thanks

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20
=20
=20

=20 From: Ravi teja ch n v [mailto:= raviteja.chnv@huawei.com]
=20 Sent: Wednesday, April 04, 2012 4:35 PM
=20 To: mapreduce-user@hadoop.apache.org
=20 Subject: RE: Calling one MR job within another MR job=20

=20 =C2=A0

=20

=20 =C2=A0

=20
=20
=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20
=20

=20 Hi Stuti,

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 If you are looking for MRjob2 to run after MRj= ob1, ie the job dependency,

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 you can use JobControl API, where you can mana= ge the dependencies.

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 Calling another Job from a Mapper is not a goo= d idea.

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 Thanks,

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 Ravi Teja

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20
=20
= =20
=20
=20
=20

=20 From: = Stuti Awasthi [stutiawasthi@hcl.com]
=20 Sent: 04 April 2012 16:04:19
=20 To: mapreduce-user@hadoop.apache.org
=20 Subject: Calling one MR job within another MR job

= =20

=20 =C2=A0

=20

=20 =C2=A0

=20
=20
=20
=20

=20 Hi all,

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 We have a usecase in which I start with= first MR1 job with input file as File1.txt, and from this job, call anothe= r MR2 job with input as File2.txt

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 So :

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 MRjob1{

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 Map(){

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 MRJob2(File2.txt)

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 }

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 }

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 MRJob2{

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 Processing=E2=80=A6.

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 }

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 My queries are is this kind of approach= is possible and how much are the implications from the performance perspec= tive.

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 Regards,

= =20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 Stuti Awasthi

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 HCL Comnet Systems a= nd Services Ltd

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 F-8/9 Basement, Sec-= 3,Noida.

=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20  

=20

=20 =C2=A0

=20

=20 =C2=A0

=20
=20

=20 =C2=A0

=20

=20 =C2=A0

=20

=20 =C2=A0

=20
=20
=20
=20

=20 ::DISCLAIMER::
=20 --------------------------------------------------------------------= ---------------------------------------------------
=20
=20 The contents of this e-mail and any attachment(s) are confidential a= nd intended for the named recipient(s) only.
=20 It shall not attach any liability on the originator or HCL or its af= filiates. Any views or opinions presented in
=20 this email are solely those of the author and may not necessarily re= flect the opinions of HCL or its affiliates.
=20 Any form of reproduction, dissemination, copying, disclosure, modifi= cation, distribution and / or publication of
=20 this message without the prior written consent of the author of this= e-mail is strictly prohibited. If you have
=20 received this email in error please delete it and notify the sender = immediately. Before opening any mail and
=20 attachments please check them for viruses and defect.
=20
=20 --------------------------------------------------------------------= ---------------------------------------------------

=20

=20 =C2=A0

=20

=20 =C2=A0

=20
=20
=20
=20
=20
=20
=20

=20 =C2=A0

=20
=20
=20
=20
=20   --========GMXBoundary27681333543628550753--