hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Combiner function
Date Mon, 02 Aug 2010 20:34:46 GMT
On Mon, Aug 2, 2010 at 4:28 PM, Jackob Carlsson
<jackob.carlsson@gmail.com> wrote:
> Thanks Nick, but "in-memory" means a combiner can only be used over a single
> mapper?right?! Is there a way we use it for several mappers as well? Also
> what do you mean by "it may or may not run on a particular map attempt"?
>
> Br,
> Jackob
>
> On Mon, Aug 2, 2010 at 5:43 PM, Nick Jones <nick.jones@amd.com> wrote:
>
>> Hi Jackob,
>> A combiner acts a lot like a reduce step but it's executed on the mapper
>> with in-memory data.  I've seen a reduction in job execution time by adding
>> one.  The one caveat to keep in mind is that it may or may not run on a
>> particular map attempt.
>>
>> Nick
>>
>>
>>
>> On 8/2/2010 10:39 AM, Jackob Carlsson wrote:
>>
>>> Hi everyone,
>>> Could anyone please help me to understand the function of combiner?
>>>
>>> Thanks in advance
>>> Jackob
>>>
>>>
>>
>>
>

> Is there a way we use it for several mappers as well?
No. That is the exact opposite goal of the combiner. It runs locally.
>it may or may not run on a particular map attempt
It only runs when certain thresholds in the framework are reached.

http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/

Mime
View raw message