hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mag Gam <magaw...@gmail.com>
Subject Re: rack awareness help
Date Thu, 04 Mar 2010 23:39:10 GMT
Thanks everyone for explaining this to me instead of giving me RTFM!

I will play around with it and see how far I get.



On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <stevel@apache.org> wrote:
> Allen Wittenauer wrote:
>>
>> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
>>
>>> Thanks Alan! Your presentation is very nice!
>>
>> Thanks. :)
>>
>>> "If you don't provide a script for rack awareness, it treats every
>>> node as if it was its own rack". I am using the default settings and
>>> the report still says only 1 rack.
>>
>> Let's take a different approach to convince you. :)
>>
>> Think about the question:  Is there a difference between all nodes in one
>> rack vs. every node acting as a lone rack?
>>
>> The answer is no, there isn't any difference.  In both cases, all copies
>> of
>> the blocks can go to pretty much any node. When a MR job runs, every node
>> would either be considered 'off rack' or 'rack-local'.
>>
>> So there is no difference.
>>
>>
>>> Do you mind sharing a script with us on how you determine a rack? and
>>> a sample <configuration> </configuration> syntax?
>>
>> Michael has already posted his, so I'll skip this one. :)
>>
>
> Think Mag probably wanted a shell script.
>
> Mag, give your machines IPv4 addresses that map to rack number. 10.1.1.* for
> rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP address by
> the top bytes, returning "10.1.1" for everything in rack one, "10.1.2" for
> rack 2; Hadoop will be happy
>

Mime
View raw message