hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikas Jadhav <vikascjadha...@gmail.com>
Subject Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Date Wed, 13 Mar 2013 08:59:20 GMT
Hi
I am specifying requirement again with example.



I have use case where i need to shufffle same (key,value) pair to multiple
reducers


For Example  we have pair  (1,"ABC") and two reducers (reducer0 and
reducer1) are there then

by default this pair will go to reduce1 (cause  (key % numOfReducer) =
(1%2) )


how i should shuffle this pair to both reducer.

Also I willing to change the code of hadoop framework if Necessory.

 Thank you

On Wed, Mar 13, 2013 at 12:51 PM, feng lu <amuseme.lu@gmail.com> wrote:

> Hi
>
> you can use Job#setNumReduceTasks(int tasks) method to set the number of
> reducer to output.
>
>
> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vikascjadhav87@gmail.com>wrote:
>
>> Hello,
>>
>> As by default Hadoop framework can shuffle (key,value) pair to only one
>> reducer
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>> Also I  willing to change the code of hadoop framework if Necessory.
>>
>>
>> Thank you
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Don't Grow Old, Grow Up... :-)
>



-- 
*
*
*

Thanx and Regards*
* Vikas Jadhav*

Mime
View raw message