hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Donofrio (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2002) MRUnit driver classes should provide ability to set a configuration object to be passed into the mapper/reducer
Date Thu, 15 Mar 2012 07:45:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229960#comment-13229960
] 

Jim Donofrio commented on MAPREDUCE-2002:
-----------------------------------------

This should be closed as a dupe, MRUNIT already does this from MAPREDUCE-1569 as Aaron said.
MRUNIT-86 also just added the withConfiguration to the 4 mapred classes: MapDriver, ReduceDriver,
MapReduceDriver, PipelineMapReduceDriver as well as fixing the conf to get pushed down to
the individual mapper/reducer in MapReduceDriver or the MapReduceDriver's in PipelineMapReduceDriver
                
> MRUnit driver classes should provide ability to set a configuration object to be passed
into the mapper/reducer
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2002
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2002
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/mrunit
>    Affects Versions: 0.20.2
>            Reporter: David Rosenstrauch
>            Priority: Minor
>
> Short description:
> Enhance the org.apache.hadoop.mrunit.mapreduce.MapDriver, ReduceDriver, and MapReduceDriver
unit test driver classes to contain "setConfiguration" and "withConfiguration" methods for
passing in user-supplied org.apache.hadoop.conf.Configuration objects, and have those configuration
objects eventually get passed on to the Context objects that are passed in to the mapper/reducer
"setup" methods.  (Rather than passing in an empty Configuration object, as is being done
now.)
> Long description:
> The MRUnit driver classes (i.e., MapDriver, ReduceDriver, and MapReduceDriver) ought
to be enhanced to contain methods for setting a Configuration object to be used by the mapper/reducer
being tested - i.e., setConfiguration() and withConfiguration().
> The only way to effectively pass parameters into a mapper or reducer is by setting properties
on a configuration object, which the mapper/reducer can then retrieve in their "setup" step,
and use to customize its operation.  As a result, specific mappers/reducers may require the
presence of specific configuration properties/parameters in order to function correctly (or
at all).  (I am currently coding such a reducer right now.)
> Testing such a mapper/reducer thus requires that the unit testing framework used provide
the ability to pass in user-supplied Configuration objects to them so that they can be tested
with appropriate parameter values.  However, MRUnit currently does not provide this ability.
 (All mappers/reducers are always passed an empty configuration object.)  And there is not
even currently any (easy) way for the end-user to fix this problem by creating a simple sub-class
that supplies this functionality, as such subclasses would require a substantial reimplementation/override
of several MRUnit framework classes.
> I believe this something that is not too difficult to fix in the MRUnit framework code,
however, and would greatly help the usability of MRUnit.
> Although I don't have time to code this enhancement right now, if needed/preferred I
could squeeze out some time to code up a patch for this.  If that's needed, please let me
know.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message