hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR
Date Thu, 05 May 2011 09:46:03 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029254#comment-13029254
] 

Steve Loughran commented on MAPREDUCE-2454:
-------------------------------------------

I'm also worried about the exception signatures of the interface. One of the hardest things
to do in any Java plugin API is getting the list of possible exceptions throw right -if not
you end up wrapping everything up. 

Presumably the list of exceptions thrown is derived from your (private) implementation. For
example, the {{initialize()}} methods throw {{ IOException, ClassNotFoundException, UnsupportedOperationException
}}.

A {{ClassNotFoundException}} ClassCastException implies the implementation is possibly playing
with classloaders -which is a dangerous game and one in which most Java developers, myself
included- shouldn't be doing. At the very least the exception list should include

{{ClassCastException}} -when the class loads but is the wrong type
{{InstantiationException}} you can't instantiate the class as its constructor isn't there
{{NoClassDefFoundError}} can't find the class though it used to be there
{{LinkageError}} etc. 

The other signatures may work, but the current initialize code is clearly biased towards one
single implementation. We'd need more implementation, and some test cases that simulate other
failures, to make sure whatever changes go into the MR engine actually handle them, presumably
by catching and failing the job. I don't see any point in trying to fall back to another sorter
as that would just hide a problem.

> Allow external sorter plugin for MR
> -----------------------------------
>
>                 Key: MAPREDUCE-2454
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Mariappan Asokan
>            Priority: Minor
>         Attachments: KeyValueIterator.java, MapOutputSorter.java, MapOutputSorterAbstract.java,
ReduceInputSorter.java
>
>
> Define interfaces and some abstract classes in the Hadoop framework to facilitate external
sorter plugins both on the Map and Reduce sides.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message