hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (MAPREDUCE-2007) Is it possible that use ArrayList or other type instead Iterable when use reduce(Object, Iterable, Context)?
Date Thu, 12 Aug 2010 18:03:17 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Owen O'Malley resolved MAPREDUCE-2007.
--------------------------------------

    Resolution: Won't Fix

The framework can't assume that all of the values fit into memory, so it is not possible to
make the API require a List object.

If you are just counting values, you should consider replacing the value with an integer and
implement a combiner that adds the counts together. It will be much more efficient. Look at
the word count example for an example of how to do this.

If you just need the first N values, just iterate through the values you need and return from
the reduce method. There is no need to exhaust the iterator.

> Is it possible that use ArrayList or other type  instead Iterable  when use reduce(Object,
Iterable, Context)?
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2007
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2007
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Hui Wen Han
>             Fix For: 0.20.2
>
>
> 1) Sometimes We only need get the elements count of the input values of Reducer task,
> but we have to iterate all the input values to calculate it.
> 2) Sometimes We only need get a few elements (for example top n,last n ,or random ) from
 the input values of Reducer task,
> if it can use ArrayList or other type  instead Iterable  when use reduce(Object, Iterable,
Context),it 's more conveniency.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message