Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com
 designates 209.85.160.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=DjgFDP9OLz9REkS5LbJEsVNhCCmvzuAyDNaGIq84/lXTuyswJM7fL+V/B5lXetAxbo
         574D1prwrNzP60N8ddz2koPsPUU8iTobZlEavf07OsHr4+0k5RorCqZq3Sgcw/FHb7kK
         QDt6QwqfSAnLsi9GOVmAx5XGS07Sy3bnOkEsY=
MIME-Version: 1.0
In-Reply-To: <126335.8536.qm@web112111.mail.gq1.yahoo.com>
References: <126335.8536.qm@web112111.mail.gq1.yahoo.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Mon, 22 Nov 2010 22:22:06 +0530
Message-ID: <AANLkTimad3DwTUfozirAzFFX_WEj5eXyKwDwsjB0j1Ju@mail.gmail.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

On Mon, Nov 22, 2010 at 10:02 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
> Hi all,
>
> I have troubles in understanding what exactly a slot is. Always we are ta=
lking about tasks assigned to slots, but I did not found anywhere what exac=
tly a slot is. I assume it represent some allocation of RAM memory as well =
as with some computation power.
>
> However, can somebody explain me what exactly a slot means (in terms of r=
esources allocated for a slot) and how this mapping(between slot and physic=
al resources) is done in Hadoop ? Or give me some hints about the files in =
the Hadoop=A0 where it may should be ?

A slot is of two types -- Map slot and Reduce slot. A slot represents
an ability to run one of these "Tasks" (map/reduce tasks) individually
at a point of time. Therefore, multiple slots on a TaskTracker means
multiple "Tasks" may execute in parallel.

Right now total slots in a TaskTracker is =3D=3D
mapred.tasktracker.map.tasks.maximum for Maps and
mapred.tasktracker.reduce.tasks.maximum for Reduces.

Hadoop is indeed trying to go towards the dynamic slot concept, which
could rely on the current resources available on a system, but work
for this is still in conceptual phases. TaskTrackers emit system
status (like CPU load, utilization, memory available/user, load
averages) in their heartbeats today (and is utilized by certain
schedulers, I think Capacity Scheduler uses it to determine stuff),
but the concept of slots is still fixed as a maximum to the above two
configurations on each TaskTracker.

For code on how slots are checked/utilized, see any Scheduler plugin's
code -- LimitTasksPerJobTaskScheduler, CapacityTaskScheduler for
example.

>
> Thanks a lot,
> Robert
>
>
>


--=20
Harsh J
www.harshj.com