hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan LeCompte" <lecom...@gmail.com>
Subject Re: Question regarding reduce tasks
Date Mon, 03 Nov 2008 16:38:14 GMT
What happens when the reducer task gets invoked more than once? My
guess is once a reducer task finishes writing the data for a
particular key to HDFS, it won't somehow get re-executed again for the
same key right?


On Mon, Nov 3, 2008 at 11:28 AM, Miles Osborne <miles@inf.ed.ac.uk> wrote:
> you can't guarantee that a reducer (or mapper for that matter) will be
> executed exactly once unless you turn-off preemptive scheduling.  but,
> a distinct key gets sent to a single reducer, so yes, only one reducer
> will see a particulat key + associated values
>
> Miles
>
> 2008/11/3 Ryan LeCompte <lecompte@gmail.com>:
>> Hello,
>>
>> Is it safe to assume that only one reduce task will ever operate on
>> values for a particular key? Or is it possible that more than one
>> reduce task can work on values for the same key? The reason I ask is
>> because I want to ensure that a piece of code that I write at the end
>> of my reducer method will only ever be executed once after all values
>> for a particular key are aggregated/summed.
>>
>> Thanks,
>> Ryan
>>
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>

Mime
View raw message