Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4B2FDD7C5 for ; Sun, 23 Sep 2012 13:31:46 +0000 (UTC) Received: (qmail 37601 invoked by uid 500); 23 Sep 2012 13:31:41 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 37485 invoked by uid 500); 23 Sep 2012 13:31:40 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 37471 invoked by uid 99); 23 Sep 2012 13:31:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Sep 2012 13:31:40 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dechouxb@gmail.com designates 209.85.216.176 as permitted sender) Received: from [209.85.216.176] (HELO mail-qc0-f176.google.com) (209.85.216.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Sep 2012 13:31:33 +0000 Received: by qcon41 with SMTP id n41so1601451qco.35 for ; Sun, 23 Sep 2012 06:31:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=KyXlY9ps6XgpVKtTMigxuzeDqUjVk9s6XJzt8s4EvKE=; b=bqV9kb2Fe40Qibf/S21SYcYeaXuG6j1H+3/dmqCDo7otvV6T3l5mkr3zVNR3UIXU6T abTyZp4ptgrOfR9yzBx1c7zcbjr8QD39zUPDoJ39kfUbzunHQZFV4PyrrfB2OXh8i7Q0 x6pDN0cN+hkD7bGYK8jZNU1/kSznISDmgvqdHkSJ611eghvO5/wWuY1B9KxPhevepbYM PqiEpdOaUPG6dKHIOJ1sSDmdU8vkiKWm2j4UCvOm/Puu7h2QeD1F5gNYgL77kse8eNE8 ndx564qldy2Fsm7wYsadOulF/YcpVrmVvTj5kYX3Nvh+I9vphcGo3AAr1TMaWugAp7m3 vjng== MIME-Version: 1.0 Received: by 10.224.70.138 with SMTP id d10mr25743536qaj.12.1348407072608; Sun, 23 Sep 2012 06:31:12 -0700 (PDT) Received: by 10.49.71.231 with HTTP; Sun, 23 Sep 2012 06:31:12 -0700 (PDT) In-Reply-To: References: Date: Sun, 23 Sep 2012 15:31:12 +0200 Message-ID: Subject: Re: How to set 2mappers on 1 job From: Bertrand Dechoux To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=bcaec51a81ec9352e204ca5e7c77 --bcaec51a81ec9352e204ca5e7c77 Content-Type: text/plain; charset=ISO-8859-1 Harsh's solution is indeed cleaner and must be what you were looking for (and there is a version for both mapred and mapreduce packages). If you are curious, see : https://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.3/src/mapred/org/apache/hadoop/mapred/lib/MultipleInputs.java https://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.3/src/mapred/org/apache/hadoop/mapred/lib/DelegatingMapper.java https://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.3/src/mapred/org/apache/hadoop/mapreduce/lib/input/MultipleInputs.java https://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.3/src/mapred/org/apache/hadoop/mapreduce/lib/input/DelegatingMapper.java Regards Bertrand On Sun, Sep 23, 2012 at 7:37 AM, Harsh J wrote: > Hi, > > There's an easier way to do what Bertrand has suggested. Look at > MultipleInputs class: > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/MultipleInputs.htmland see this blog post for an example on how to use it: > http://kickstarthadoop.blogspot.in/2011/09/joins-with-plain-map-reduce.html > > Note though that the reducer input key and value types are singular, and > you need to ensure that. There's no easy way around that aside of using > generic containers. > > > On Sun, Sep 23, 2012 at 9:34 AM, kumudu harshani > wrote: > >> I am sorry.. I didn't get you.. shouldn't i handle that with jobconf code. >> >> The confusion i have is, if i put like: >> >> JobConf conf2 = new JobConf(WordCount.class); >> Job job2 = new Job(conf2); >> conf2.setOutputKeyClass(IntWritable.class); >> conf2.setOutputValueClass(Text.class); >> >> conf2.setMapperClass(Map1.class); >> conf2.setReducerClass(Reduce1.class); >> >> ---it will execute Map1.class and then Reduce1.class. >> >> so if i have Mapper1a.class and Mapper2a.class, how should i write the >> code of job to execute both and then execute Reducer.class such that, >> Reducer will take both mappers (1a, 1b) emit outputs... >> >> thanks >> kumudu >> >> On Sun, Sep 23, 2012 at 9:23 AM, Bertrand Dechoux wrote: >> >>> You can use the map.input.file property to decide which logic should >>> your mapper apply. >>> Regards >>> Bertrand >>> >>> >>> On Sun, Sep 23, 2012 at 5:40 AM, kumudu harshani < >>> kumuduharshani@gmail.com> wrote: >>> >>>> Hi... >>>> Could someone help me with following scenario.. >>>> >>>> I want implement a job which should get 2 mapper outputs and send them >>>> to 1 reducer. Attached image show the flow I wanted.... >>>> >>>> >>>> >>>> >>>> Normal flow is like: >>>> >>>> JobConf conf2 = new JobConf(WordCount.class); >>>> Job job2 = new Job(conf2); >>>> conf2.setOutputKeyClass(IntWritable.class); >>>> conf2.setOutputValueClass(Text.class); >>>> >>>> conf2.setMapperClass(Map1.class); >>>> conf2.setReducerClass(Reduce1.class); >>>> >>>> --- where it takes 1 mapper and 1 reducer. What i want is to set 2 >>>> maps(mapper1a, mapper1b) and 1 reducer... >>>> Is that possible, if so could someone please help.. >>>> >>>> thanks >>>> kumudu >>>> -- >>>> >>>> *Kumudu Samarappuli* | Creative Search Technologies, a Microsoft IEG >>>> Partner | Software Engineer I | m: +94 719 258 242 | >>>> www.microsoft.com/enterprisesearch >>>> >>>> >>> >>> >>> -- >>> Bertrand Dechoux >>> >> >> >> >> -- >> >> *Kumudu Samarappuli* | Creative Search Technologies, a Microsoft IEG >> Partner | Software Engineer I | m: +94 719 258 242 | >> www.microsoft.com/enterprisesearch >> >> > > > -- > Harsh J > -- Bertrand Dechoux --bcaec51a81ec9352e204ca5e7c77 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Harsh's solution is indeed cleaner and must be what you were looking fo= r (and there is a version for both mapred and mapreduce packages).

=
If you are curious, see :


Regards

Bertrand

On Sun, Sep 23, 2012 at 7:37 AM, Harsh J <harsh@cloudera.com> wrote:
Hi,

There's an easier= way to do what Bertrand has suggested. Look at MultipleInputs class:=A0http://hadoop.ap= ache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/Mult= ipleInputs.html and see this blog post for an example on how to use it:= =A0http://kickstarthadoop.blogspot.in/2011/0= 9/joins-with-plain-map-reduce.html

Note though that the reducer input key and value types = are singular, and you need to ensure that. There's no easy way around t= hat aside of using generic containers.


On Sun, Sep 23, 2012 at 9:34 AM, kumudu harshani <kumuduharshani@gm= ail.com> wrote:
I am sorry.. I didn't get you.. shouldn't i handle that with jobcon= f code.

The confusion i have is, if i put like:

JobConf conf2 =3D new JobConf(WordCount.class);
Job job2 =3D new Job(conf2);
conf2.setOutputKeyClass(IntWritable.class);
conf2.setOutputValueClass(Te= xt.class);

conf2.setMapperClass(Map1.class);
conf2.setReducerClass(Reduce1.clas= s);

=A0---it will execute Map1.class and then Reduce1.c= lass.

so if i have Mapper1a.class and Mapper2a.class, how should i w= rite the code of job to execute both and then execute Reducer.class such th= at, Reducer will take both mappers (1a, 1b) emit outputs...

thanks
kumudu
<= div>
On Sun, Sep 23, 2012 at 9:23 AM, Bertran= d Dechoux <dechouxb@gmail.com> wrote:
You can use the=A0map.input.file property to decide which logic should your= mapper apply.
Regards
Bertrand


On Sun, Sep 23, 2012 at 5:40 AM, kumudu harshani <= kumuduharshani@gmail.com> wrote:
Hi...
Could someone help me with followin= g scenario..

I want implement a job which should get 2 mapper output= s and send them to 1 reducer. Attached image show the flow I wanted....

3D""


Normal flow is like:

JobC= onf conf2 =3D new JobConf(WordCount.class);
Job job2 =3D new Job(conf2);=
conf2.setOutputKeyClass(IntWritable.class);
conf2.setOutputValueClas= s(Text.class);

conf2.setMapperClass(Map1.class);
conf2.setReducerClass(Reduce1.clas= s);

--- where it takes 1 mapper and 1 reducer. What i want is= to set 2 maps(mapper1a, mapper1b) and 1 reducer...
Is that possible, i= f so could someone please help..

thanks
kumudu
--

= K= umudu Samarappuli | Creative Search Technologies, a = Microsoft IEG Partner | Software Engineer I | m: +94 719 258 242 | www.microsof= t.com/enterprisesearch





--
Bertrand Dechoux



--

Kumudu Samarappuli | Creative Search Technologies, a Microsoft IEG Partner | Software Engineer I | m: +94 719 258 242 | www.microsof= t.com/enterprisesearch





<= /div>--
Harsh J


--
Bertrand Dec= houx
--bcaec51a81ec9352e204ca5e7c77--