hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Kimball <aa...@cloudera.com>
Subject Re: number of reducers
Date Sun, 06 Jun 2010 23:55:14 GMT
Yes. LJR sets your number of reduce tasks to 1 if the number is >= 1.

Subnote: I've posted a patch to fix this at MAPREDUCE-434, but it's not
committed.

- Aaron

On Mon, Jun 7, 2010 at 1:42 AM, Torsten Curdt <tcurdt@vafer.org> wrote:

> I see only one.
>
> Could it be that using the LocalJobRunner interferes here?
>
> On Mon, Jun 7, 2010 at 01:31, Eric Sammer <esammer@cloudera.com> wrote:
> > Torsten:
> >
> > To clarify, how many reducers do you actually see? (i.e. Do you see 4
> > reducers or 1?) It should work as you expect.
> >
> > On Sun, Jun 6, 2010 at 1:33 PM, Torsten Curdt <tcurdt@vafer.org> wrote:
> >> When I set
> >>
> >>  job.setPartitionerClass(MyPartitioner.class);
> >>  job.setNumReduceTasks(4);
> >>
> >> I would expect to see my MyParitioner get called with
> >>
> >>  getPartition(key, value, 4)
> >>
> >> but still I see it only get called with 1.
> >>
> >> If also tried setting
> >>
> >>                conf.set("mapred.map.tasks.speculative.execution",
> "false");
> >>                conf.set("mapred.reduce.tasks.speculative.execution",
> "false");
> >>
> >> What am I missing here?
> >>
> >> cheers
> >> --
> >> Torsten
> >>
> >
> >
> >
> > --
> > Eric Sammer
> > phone: +1-917-287-2675
> > twitter: esammer
> > data: www.cloudera.com
> >
>

Mime
View raw message