hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
Date Fri, 16 Nov 2012 18:36:16 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arun C Murthy updated MAPREDUCE-2454:
-------------------------------------

    Status: Open  (was: Patch Available)

Ok, I'm glad I caught this...

One issue is that you don't want to pass a shuffle down to the merger, it's the other way
around - you want to pass a merger to the shuffle.

Also, I'm not wild about making the change to MapOutput to shuffle itself, like I said - it
was initially designed as merely a 'struct'.

----

I've said this before on this jira, I'd really appreciate if you could break this apart into
smaller chunks - it makes a reviewer's job much easier... for e.g. I missed the shuffle/merger
change since it's a largish patch.

Can you pls create some natural sub-tasks:
# Move MapOutputCollector out as an interface into a new class outside of MapTask
# Introduce a new ReduceInputMerger or some such interface which is sufficient for your purposes
# Then mark all interfaces you need as 'LimitedPrivate' 

I appreciate your patience, thanks again.
                
> Allow external sorter plugin for MR
> -----------------------------------
>
>                 Key: MAPREDUCE-2454
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>    Affects Versions: 2.0.2-alpha, 2.0.0-alpha, 3.0.0
>            Reporter: Mariappan Asokan
>            Assignee: Mariappan Asokan
>            Priority: Minor
>              Labels: features, performance, plugin, sort
>         Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, KeyValueIterator.java,
MapOutputSorterAbstract.java, MapOutputSorter.java, mapreduce-2454-modified-code.patch, mapreduce-2454-modified-test.patch,
mapreduce-2454-new-test.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454-protection-change.patch,
mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, ReduceInputSorter.java
>
>
> Define interfaces and some abstract classes in the Hadoop framework to facilitate external
sorter plugins both on the Map and Reduce sides.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message