Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MailScanner-NULL-Check: 1233224222.57393@Fym7eC/yaOsUz7YnPzUeVQ
Message-ID: <4978479C.805@apache.org>
Date: Thu, 22 Jan 2009 10:17:00 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.19 (X11/20081209)
MIME-Version: 1.0
To: core-user@hadoop.apache.org
Subject: Re: running hadoop on heterogeneous hardware
References: <3b5f72030901211427q4b033446ofb248afa6d39a1c4@mail.gmail.com>
In-Reply-To: <3b5f72030901211427q4b033446ofb248afa6d39a1c4@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Bill Au wrote:
> Is hadoop designed to run on homogeneous hardware only, or does it work just
> as well on heterogeneous hardware as well?  If the datanodes have different
> disk capacities, does HDFS still spread the data blocks equally amount all
> the datanodes, or will the datanodes with high disk capacity end up storing
> more data blocks?  Similarily, if the tasktrackres have different numbers of
> CPUs, is there a way to configure hadoop to run more tasks on those
> tasktrackers that have more CPUs?  Is that simply a matter of setting
> mapred.tasktracker.map.tasks.maximum and
> mapred.tasktracker.reduce.tasks.maximum differently on the tasktrackers?
> 
> Bill
> 

Life is simpler on homogenous boxes; by setting the maximum tasks 
differently for the different machines, you do limit the amount of work 
that gets pushed out to those boxes. More troublesome is slower 
CPUs/HDDs, they arent picked up directly, though speculative work can 
handle some of this

One interesting bit of research would be something adaptive; something 
to monitor throughput and tune those values based on performance; that 
would detect variations in a cluster and work with with it, rather than 
requiring you  to know the capabilities of every machine.

-steve