hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torsten Curdt <tcu...@vafer.org>
Subject Re: number of reducers
Date Mon, 07 Jun 2010 00:11:31 GMT
Great. Just thought I'd be missing something :)

On Mon, Jun 7, 2010 at 01:55, Aaron Kimball <aaron@cloudera.com> wrote:
> Yes. LJR sets your number of reduce tasks to 1 if the number is >= 1.
> Subnote: I've posted a patch to fix this at MAPREDUCE-434, but it's not
> committed.
> - Aaron
>
> On Mon, Jun 7, 2010 at 1:42 AM, Torsten Curdt <tcurdt@vafer.org> wrote:
>>
>> I see only one.
>>
>> Could it be that using the LocalJobRunner interferes here?
>>
>> On Mon, Jun 7, 2010 at 01:31, Eric Sammer <esammer@cloudera.com> wrote:
>> > Torsten:
>> >
>> > To clarify, how many reducers do you actually see? (i.e. Do you see 4
>> > reducers or 1?) It should work as you expect.
>> >
>> > On Sun, Jun 6, 2010 at 1:33 PM, Torsten Curdt <tcurdt@vafer.org> wrote:
>> >> When I set
>> >>
>> >>  job.setPartitionerClass(MyPartitioner.class);
>> >>  job.setNumReduceTasks(4);
>> >>
>> >> I would expect to see my MyParitioner get called with
>> >>
>> >>  getPartition(key, value, 4)
>> >>
>> >> but still I see it only get called with 1.
>> >>
>> >> If also tried setting
>> >>
>> >>                conf.set("mapred.map.tasks.speculative.execution",
>> >> "false");
>> >>                conf.set("mapred.reduce.tasks.speculative.execution",
>> >> "false");
>> >>
>> >> What am I missing here?
>> >>
>> >> cheers
>> >> --
>> >> Torsten
>> >>
>> >
>> >
>> >
>> > --
>> > Eric Sammer
>> > phone: +1-917-287-2675
>> > twitter: esammer
>> > data: www.cloudera.com
>> >
>
>

Mime
View raw message