hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: Hadoop - how exactly is a slot defined
Date Wed, 24 Nov 2010 17:32:47 GMT

On Wed, Nov 24, 2010 at 10:23 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
> Hi,
> I am sorry bothering again about this subject, but still I am not very convinced what
Hadoop assumes a slot is. I understood it represent smth in terms of CPU/Memory, so you have
to allocate corresponding numbers of map/reduce slots based on your configurations.
> BUT, I cannot understand yet, if Hadoop make any mapping between the concept of slot
and physical resources itself, or are just some numbers and you can go over only with this

The slot amount is the user's homework for now.

> I looked on the code, but I am not able to figure out if Hadoop really did some checking
between number of slots and physical resources, or just is limited by the 2 numbers(for maximum
number of map slots and reduce slots) and play with this numbers only. That means, the user
should give more interpretation of what a slot really may be: (Only one slot per core, one
slot per 512 MB, etc) when configure the number of map/reduce slots on his machines.

Yes, Hadoop does not dynamically detect any such thing yet. The setup
is ignorant to a machine's hardware and blindly relies on the
configurations passed at start up.

I usually set M = No. of CPUs + 1, and R = Prime nearest No. of CPUs.
But needs may vary depending on the nature of jobs it is going to
perform; sometimes you may need lesser CPU but more Memory/Task, so
configure based on your application knowledge.

Harsh J

View raw message