hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: rack awareness help
Date Thu, 04 Mar 2010 14:21:00 GMT
Allen Wittenauer wrote:
> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
> 
>> Thanks Alan! Your presentation is very nice!
> 
> Thanks. :)
> 
>> "If you don't provide a script for rack awareness, it treats every
>> node as if it was its own rack". I am using the default settings and
>> the report still says only 1 rack.
> 
> Let's take a different approach to convince you. :)
> 
> Think about the question:  Is there a difference between all nodes in one
> rack vs. every node acting as a lone rack?
> 
> The answer is no, there isn't any difference.  In both cases, all copies of
> the blocks can go to pretty much any node. When a MR job runs, every node
> would either be considered 'off rack' or 'rack-local'.
> 
> So there is no difference.
> 
> 
>> Do you mind sharing a script with us on how you determine a rack? and
>> a sample <configuration> </configuration> syntax?
> 
> Michael has already posted his, so I'll skip this one. :)
> 

Think Mag probably wanted a shell script.

Mag, give your machines IPv4 addresses that map to rack number. 10.1.1.* 
for rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP 
address by the top bytes, returning "10.1.1" for everything in rack one, 
"10.1.2" for rack 2; Hadoop will be happy

Mime
View raw message