hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From samir das mohapatra <samir.help...@gmail.com>
Subject Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Date Wed, 13 Mar 2013 09:02:00 GMT
Use can use Custom Partitioner for that same.



Regards,

Samir.


On Wed, Mar 13, 2013 at 2:29 PM, Vikas Jadhav <vikascjadhav87@gmail.com>wrote:

>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example  we have pair  (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause  (key % numOfReducer) =
> (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
>  Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <amuseme.lu@gmail.com> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vikascjadhav87@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I  willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>

Mime
View raw message