Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1E9BAD1D9 for ; Wed, 13 Mar 2013 09:10:50 +0000 (UTC) Received: (qmail 63673 invoked by uid 500); 13 Mar 2013 09:10:45 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 63453 invoked by uid 500); 13 Mar 2013 09:10:45 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 63443 invoked by uid 99); 13 Mar 2013 09:10:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Mar 2013 09:10:44 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Ajay.Srivastava@guavus.com designates 204.232.241.167 as permitted sender) Received: from [204.232.241.167] (HELO mx1.guavus.com) (204.232.241.167) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Mar 2013 09:10:37 +0000 Received: from mx1.guavus.com ([204.232.241.167]) by mx1.guavus.com ([204.232.241.167]) with mapi id 14.01.0379.000; Wed, 13 Mar 2013 02:10:15 -0700 From: Ajay Srivastava To: "" Subject: Re: How to shuffle (Key,Value) pair from mapper to multiple reducer Thread-Topic: How to shuffle (Key,Value) pair from mapper to multiple reducer Thread-Index: AQHOH7IzzB0BoFXY00GFuG0gQczUPpijrM0AgAAbQACAAAMKgA== Date: Wed, 13 Mar 2013 09:10:15 +0000 Message-ID: <340AEACF-E082-46EC-91BB-B4457438E91C@guavus.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [61.12.3.109] Content-Type: multipart/alternative; boundary="_000_340AEACFE08246EC91BBB4457438E91Cguavuscom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_340AEACFE08246EC91BBB4457438E91Cguavuscom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Emit (key, value) twice from mapper by modifying key as key' =3D (key, part= Id) and record becomes (key', value) >From custom partitioner, send record to reducer based on partId. Ignore par= tId field in reducer. Regards, Ajay Srivastava On 13-Mar-2013, at 2:29 PM, Vikas Jadhav wrote: Hi I am specifying requirement again with example. I have use case where i need to shufffle same (key,value) pair to multiple = reducers For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer= 1) are there then by default this pair will go to reduce1 (cause (key % numOfReducer) =3D (1= %2) ) how i should shuffle this pair to both reducer. Also I willing to change the code of hadoop framework if Necessory. Thank you On Wed, Mar 13, 2013 at 12:51 PM, feng lu > wrote: Hi you can use Job#setNumReduceTasks(int tasks) method to set the number of re= ducer to output. On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav > wrote: Hello, As by default Hadoop framework can shuffle (key,value) pair to only one red= ucer I have use case where i need to shufffle same (key,value) pair to multiple = reducers Also I willing to change the code of hadoop framework if Necessory. Thank you -- Thanx and Regards Vikas Jadhav -- Don't Grow Old, Grow Up... :-) -- Thanx and Regards Vikas Jadhav --_000_340AEACFE08246EC91BBB4457438E91Cguavuscom_ Content-Type: text/html; charset="iso-8859-1" Content-ID: <55C288A3FFEABC458D2D3309861F9F7F@guavus.com> Content-Transfer-Encoding: quoted-printable
Emit (key, value) twice from mapper by modifying key as key' =3D (key,= partId) and record becomes (key', value)
From custom partitioner, send record to reducer based on partId. Ignor= e partId field in reducer.


Regards,
Ajay Srivastava


On 13-Mar-2013, at 2:29 PM, Vikas Jadhav wrote:

 
Hi
I am specifying requirement again with example.
 
 
 
I have use case where i need to shufffle same (key,value) pair to mult= iple reducers
 
 
For Example  we have pair  (1,"ABC") and two reduc= ers (reducer0 and reducer1) are there then
 
by default this pair will go to reduce1 (cause  (key % numOfReduc= er) =3D (1%2) )
 
 
how i should shuffle this pair to both reducer.
 
Also I willing to change the code of hadoop framework if Necessory.
 
Thank you

On Wed, Mar 13, 2013 at 12:51 PM, feng lu <amuseme.lu@gm= ail.com> wrote:
Hi

you can use Job#setNumReduceTasks(int tasks) method to set the number = of reducer to output.


On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vikascjad= hav87@gmail.com> wrote:
Hello,
 
As by default Hadoop framework can shuffle (key,value) pair to only on= e reducer
 
I have use case where i need to shufffle same (key,value) pair to mult= iple reducers
 
Also I  willing to change the code of hadoop framework if Necesso= ry.
 
 
Thank you

--


Thanx and Regards
 Vikas Jadhav



--
Don't Grow Old, Grow Up... :-)



--


Thanx and Regards
 Vikas Jadhav

--_000_340AEACFE08246EC91BBB4457438E91Cguavuscom_--