hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shrijeet Paliwal <shrij...@rocketfuel.com>
Subject Re: Is it pissible get a number of mapper tasks?
Date Sat, 04 Dec 2010 02:06:06 GMT
>>mapred.map.tasks is set for every job before launch and is the total
>>number of maps that are going to run for a successful result.

A note on mapred.map.tasks.
Hadoop does not honor mapred.map.tasksĀ all the time. It is just a hint
for the framework, actual number of map tasks launched may be
different. *I think*.

On Fri, Dec 3, 2010 at 6:01 PM, Harsh J <qwertymaniac@gmail.com> wrote:
>
> Minor correction, it is:
> mapred.tip.id is the task's id (contains various info about the task,
> map/reduce).
> mapred.task.id is the task's _attempt_ id (basically tip id, with
> attempt information, map/reduce).
>
> On Sat, Dec 4, 2010 at 7:29 AM, Harsh J <qwertymaniac@gmail.com> wrote:
> > Hi,
> >
> > (Answers may be 0.20 specific)
> >
> > On Sat, Dec 4, 2010 at 6:41 AM, Jason <urgisb@gmail.com> wrote:
> >> In my mapper code I need to know the total number of mappers which is the same
as number of input splits.
> >> (I need it for unique int Id generation)
> >
> > mapred.map.tasks is set for every job before launch and is the total
> > number of maps that are going to run for a successful result.
> >
> > There are more identifiers, however:
> > mapred.task.partition is the unique partition id per task (perhaps you
> > can utilize this integer itself, applies to map/reduce)
> >
> > (Some of these vars have methods of setter/getter in JobConf itself)
> >
> >>
> >>
> >> Basically Im looking for an analog of context.getNumReduceTasks() but can't
find it.
> >>
> >>
> >> Thanks
> >>
> >>
> >>
> >>>
> >>
> >
> >
> >
> > --
> > Harsh J
> > www.harshj.com
> >
>
>
>
> --
> Harsh J
> www.harshj.com

Mime
View raw message