hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikas Jadhav (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5063) Transfering mapper output (key,value) pair to multiple reducer
Date Mon, 25 Mar 2013 14:29:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612666#comment-13612666

Vikas Jadhav commented on MAPREDUCE-5063:

Hi Harsh,

I had also planned by this way only and currently going to implement it in
using user code but

why i thought it will be good to have separate API is

as you already know that there is lot of data reading and writing in
if we implement this using user code it is possible that it may increase
number of
writes to be done to local disk because we may have write same pair more
than one
i think there may be problem with original (key,value) pair because we may
change key -> (key_r1,value) and (key_r2,value)  here we r changing "key"
which is not desirable

so my point is that can we have approach where we can shuffle pair without
writing it two times.


Thanx and Regards*
* Vikas Jadhav*

> Transfering mapper output  (key,value) pair to multiple reducer 
> ----------------------------------------------------------------
>                 Key: MAPREDUCE-5063
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5063
>             Project: Hadoop Map/Reduce
>          Issue Type: Wish
>    Affects Versions: 1.0.3
>            Reporter: Vikas Jadhav
>            Priority: Minor
> Currently  in  Hadoop MapReduce mapper output in (key,value) form can be transfered to
only one reducer 
> Our goal is to be able transfer/shuffle (key,value) pair to multiple reducer
> Note:- we need to shuffle same pair to number of reducers

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message