hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sameer Paranjpye (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3366) Shuffle/Merge improvements
Date Tue, 13 May 2008 18:51:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596493#action_12596493
] 

Sameer Paranjpye commented on HADOOP-3366:
------------------------------------------

To clarify, we use the ramfs in exactly one place. Even there it appears to hinder a tight
implementation rather than help (i.e. we have to special case DataOutputBuffer for it to avoid
value copies, build another workaround to avoid key copies etc.). Let's not use it.

For in memory merges, an iterator over a key/value sequence in a buffer seems a much better
abstraction than an input stream.

> Shuffle/Merge improvements
> --------------------------
>
>                 Key: HADOOP-3366
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3366
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.18.0
>
>
> This is intended to be a meta-issue to track various improvements to shuffle/merge in
the reducer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message