hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-403) close method in a Mapper should be provided with OutputCollector and a Reporter
Date Sun, 13 Aug 2006 05:56:14 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-403?page=comments#action_12427732 ] 
Runping Qi commented on HADOOP-403:

In my app, I added an OutputCollector variable (and a Reporter variable if necessary) in my
Map/Reduce classes so that the close method can access them. Those variables are assigned
only once my the first call to Map/Reduce method.

If the framework should offer help here, we can introduce to Closable a method setCollector(OutputCollector)
and setReporter(Reporter). When the framework creates a Mapper/Reducer object, it calls these

> close method in a Mapper should be provided with OutputCollector and a Reporter
> -------------------------------------------------------------------------------
>                 Key: HADOOP-403
>                 URL: http://issues.apache.org/jira/browse/HADOOP-403
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.5.0
>         Environment: all
>            Reporter: Milind Bhandarkar
>         Assigned To: Milind Bhandarkar
>             Fix For: 0.6.0
> For mappers with side-effects, or mappers that work as aggregators (i.e. no output on
individual key-value pairs, but an aggregate output at the end of all key-value pairs), output
should be performed in the close method. For this purpose, we need to supply output collector
and reporter to the close method of Mapper. This involves interface change, though. Thoughts

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message