hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mariappan Asokan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR
Date Wed, 17 Oct 2012 03:22:08 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477565#comment-13477565
] 

Mariappan Asokan commented on MAPREDUCE-2454:
---------------------------------------------

 Hi Arun,
  I stepped back and looked at some of the Jiras that were developed under branches(like MAPREDUCE-279,
HDFS-1623, etc.)  They contributed major enhancements to Hadoop.  MAPREDUCE-2454 on the other
hand has just refactored existing code to make some classes more modular and pluggable.

There is no significant new functionality added.  There is no change in the way shuffling
is done.  The communication between shuffle and merge is kept intact.  The MAPREDUCE-318 refactoring
in fact helped a lot.  There is only a minor rearranging of the code in MAPREDUCE-2454.

I followed your advice diligently to work on the trunk first and I picked up suggestions from
some other developers who are watching this Jira.  During our brief meetings, the suggestions
you gave were very valuable and I followed them.

I have been testing the changes for the last year or so and I have been keeping the Jira up
to the latest trunk.  I have not hit any issues in my testing w.r.t both functionally and
performance.

I understand that your time is very precious.  I already updated the design document for you
to make it easier for code review.  If you can suggest anything that will expedite the committing,
please let me know.

Thanks.

-- Asokan
                
> Allow external sorter plugin for MR
> -----------------------------------
>
>                 Key: MAPREDUCE-2454
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>    Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha
>            Reporter: Mariappan Asokan
>            Assignee: Mariappan Asokan
>            Priority: Minor
>              Labels: features, performance, plugin, sort
>         Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, KeyValueIterator.java,
MapOutputSorterAbstract.java, MapOutputSorter.java, mapreduce-2454.patch, mapreduce-2454.patch,
mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, ReduceInputSorter.java
>
>
> Define interfaces and some abstract classes in the Hadoop framework to facilitate external
sorter plugins both on the Map and Reduce sides.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message