Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6C355D18E for ; Wed, 13 Mar 2013 09:02:08 +0000 (UTC) Received: (qmail 23180 invoked by uid 500); 13 Mar 2013 09:02:03 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 23084 invoked by uid 500); 13 Mar 2013 09:02:03 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 23063 invoked by uid 99); 13 Mar 2013 09:02:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Mar 2013 09:02:02 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dasmohap@adobe.com designates 64.18.1.185 as permitted sender) Received: from [64.18.1.185] (HELO exprod6og103.obsmtp.com) (64.18.1.185) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Mar 2013 09:01:53 +0000 Received: from outbound-smtp-1.corp.adobe.com ([192.150.11.134]) by exprod6ob103.postini.com ([64.18.5.12]) with SMTP ID DSNKUUBAbP38s5YspuuYkFGeVbg7/4BTBE5D@postini.com; Wed, 13 Mar 2013 02:01:33 PDT Received: from inner-relay-1.corp.adobe.com ([153.32.1.51]) by outbound-smtp-1.corp.adobe.com (8.12.10/8.12.10) with ESMTP id r2D8wQ1v029767 for ; Wed, 13 Mar 2013 01:58:26 -0700 (PDT) Received: from nahub02.corp.adobe.com (nahub02.corp.adobe.com [10.8.189.98]) by inner-relay-1.corp.adobe.com (8.12.10/8.12.10) with ESMTP id r2D91VAV005870 for ; Wed, 13 Mar 2013 02:01:31 -0700 (PDT) Received: from INDIACAS01.corp.adobe.com (10.40.5.6) by nahub02.corp.adobe.com (10.8.189.98) with Microsoft SMTP Server (TLS) id 8.3.298.1; Wed, 13 Mar 2013 02:01:31 -0700 Received: from indiambx02.corp.adobe.com ([10.40.5.60]) by indiacas01.corp.adobe.com ([10.40.5.6]) with mapi; Wed, 13 Mar 2013 14:31:28 +0530 From: Samir Kumar Das Mohapatra To: "user@hadoop.apache.org" Date: Wed, 13 Mar 2013 14:31:27 +0530 Subject: RE: How to shuffle (Key,Value) pair from mapper to multiple reducer Thread-Topic: How to shuffle (Key,Value) pair from mapper to multiple reducer Thread-Index: Ac4fySKAAHKEWq8zQUqDhAOB9HLd+gAAB8Kw Message-ID: <535B028B4629374D86894192BE5184DD2751533F@indiambx02.corp.adobe.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_535B028B4629374D86894192BE5184DD2751533Findiambx02corpa_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_535B028B4629374D86894192BE5184DD2751533Findiambx02corpa_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Use can use Custom Partitioner for that same. Regards, Samir. From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com] Sent: 13 March 2013 14:29 To: user@hadoop.apache.org Subject: Re: How to shuffle (Key,Value) pair from mapper to multiple reduce= r Hi I am specifying requirement again with example. I have use case where i need to shufffle same (key,value) pair to multiple = reducers For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer= 1) are there then by default this pair will go to reduce1 (cause (key % numOfReducer) =3D (1= %2) ) how i should shuffle this pair to both reducer. Also I willing to change the code of hadoop framework if Necessory. Thank you On Wed, Mar 13, 2013 at 12:51 PM, feng lu > wrote: Hi you can use Job#setNumReduceTasks(int tasks) method to set the number of re= ducer to output. On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav > wrote: Hello, As by default Hadoop framework can shuffle (key,value) pair to only one red= ucer I have use case where i need to shufffle same (key,value) pair to multiple = reducers Also I willing to change the code of hadoop framework if Necessory. Thank you -- Thanx and Regards Vikas Jadhav -- Don't Grow Old, Grow Up... :-) -- Thanx and Regards Vikas Jadhav --_000_535B028B4629374D86894192BE5184DD2751533Findiambx02corpa_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Use can u= se Custom Partitioner for that same.

 

Regards,=

Samir.<= /p>

 

= From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
Sent: 13= March 2013 14:29
To: user@hadoop.apache.org
Subject: R= e: How to shuffle (Key,Value) pair from mapper to multiple reducer

 

 

Hi

I am specifying requirement again with ex= ample.

 

<= /div>

 

 

I have use ca= se where i need to shufffle same (key,value) pair to multiple reducers=

 

<= p class=3DMsoNormal> 

Fo= r Example  we have pair  (1,"ABC") and two reducers (re= ducer0 and reducer1) are there then

 

by defaul= t this pair will go to reduce1 (cause  (key % numOfReducer) =3D (1%2) = )

 

=

 

how i should shuffle this pair to both reducer.

 

Also I willing to change the code of hadoop framework if Necessory. <= /o:p>

 

Thank you

=

On Wed, Mar 13, 2013 at 12:51 PM, feng lu &= lt;amuseme.lu@gma= il.com> wrote:

Hi<= /o:p>

 

you can use Job#setNumReduceTasks(int tasks) method to s= et the number of reducer to output.

 

=

On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <= ;vikascjadhav= 87@gmail.com> wrote:

Hello,<= o:p>

 

As by default Hadoop framework can shuffle (key,val= ue) pair to only one reducer

 

I have use case where= i need to shufffle same (key,value) pair to multiple reducers

 

Also I  willing to change the code of hadoop framework if= Necessory.

 

 

Thank you

= --

 

=  

Thanx and Regards

 Vikas Jadhav



 

--
Don't Grow Old, Grow Up... :-)




--

 

<= span style=3D'color:red'> 

Thanx and Regards

 Vikas Ja= dhav

= --_000_535B028B4629374D86894192BE5184DD2751533Findiambx02corpa_--