hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Roelofs <roel...@yahoo-inc.com>
Subject Re: conceptual question regarding slots
Date Wed, 02 Mar 2011 22:07:54 GMT
> Could someone throw some light as to how intuitively fixed-type slots in
> Hadoop have a negative impact of cluster utilization as mentioned in Arun's
> blog?

It's pretty simple:  in the course of time, a real cluster with a mix of
job types and submission times will have periods of high map-slot usage
(with reduce-slots available), and vice versa.  Being able to assign machine
resources to whatever kind of task needs them at any given time means you're
not wasting those resources.

In short:  map- vs. reduce-slots is an artificial and unnecessary restriction.


View raw message