Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.210.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <2011121212530212660634@163.com>
References: <2011121211184481597714@163.com>
 <8EB66E99-D6E9-4B11-86E0-ADCED15198F3@cloudera.com>
 <2011121212530212660634@163.com>
From: Harsh J <harsh@cloudera.com>
Date: Mon, 12 Dec 2011 10:33:02 +0530
Message-ID: 
 <CAOcnVr1w_KdoPc_mBu8FDRmbWBgFkfQZmsH9mLRDqaNUgRVVmw@mail.gmail.com>
Subject: Re: Re: About slots of tasktracker and munber of map taskers
To: mapreduce-user@hadoop.apache.org, tanjun_2525 <tanjun_2525@163.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Tan,

As an admin, I can even choose to set configuration to even 100 slots
on a 4-core node, if I feel like burning the box. There is no hardware
auto-detection, and the slot limit is entirely controlled by the
mapred-site.xml for that TaskTracker.

The book merely tries to tell that you need to set these maximum slot
settings based on your hardware knowledge on each node -- TaskTrackers
do nothing of that sort on their own.

There is some CPU/Memory considerations taken into account by a
variety of non-default Schedulers in JobTracker, but your slot limits
per tasktracker is entirely controlled by configuration.

2011/12/12 Tan Jun <tanjun_2525@163.com>:
> Hi Harsh,
> Now I know the number of maps and reduces run simultaneously is set by th=
e
> administrator in mapred-site.xml with default value 2.
> But I cant get the point about number of slots.
> For my understanding by now,
> the number of=A0 slots is decides by hardware that administrator cannot
> change.
> Is that wright?
>
> ________________________________
> Tan Jun
>
> From:=A0Harsh J
> Date:=A02011-12-12=A012:22
> To:=A0mapreduce-user
> Subject:=A0Re: About slots of tasktracker and munber of map taskers
> Hi Tan,
>
> On 12-Dec-2011, at 8:48 AM, Tan Jun wrote:
>
> Hi,
> I dont really understand the meaning of the sentences in "The Definitive
> Guide"(page 155):
>
> Tasktrackers=A0have=A0a=A0fixed=A0number=A0of=A0slots=A0for=A0map=A0tasks=
=A0and=A0for=A0reduce=A0tasks:=A0for=A0example,
> a=A0tasktracker=A0may=A0be=A0able=A0to=A0run=A0two=A0map=A0tasks=A0and=A0=
two=A0reduce=A0tasks=A0simultaneously.
> (The=A0precise=A0number=A0depends=A0on=A0the=A0number=A0of=A0cores=A0and=
=A0the=A0amount=A0of
> memory=A0on=A0the=A0tasktracker;=A0see=A0=93Memory=94=A0on=A0page=A0254.)
>
> Does that mean the=A0number of slots is fixed and the number of maps run
> simultaneously is set by user?
>
>
> Not by the user, but by the administrator. Each tasktracker is configured=
 in
> production with a 'task slot' upper limit - say, 8 maps and 4 reducers fo=
r a
> 12-core machine. This is not auto-configured (unless you use auto cluster
> setup+configuration tools that determine it for you [0]), and has to be s=
et
> when configuring Hadoop daemons.
>
> The book means to imply that you need to set these, based on the memory a=
nd
> CPU configuration of your machines. By default, tasktrackers have limits =
of
> 2+2.
>
> See=A0http://wiki.apache.org/hadoop/LimitingTaskSlotUsage
>
> [0] -=A0http://www.cloudera.com/products-services/tools/=A0is one.


--=20
Harsh J