hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Thomas <tho...@hep.caltech.edu>
Subject Re: rack awareness help
Date Thu, 04 Mar 2010 01:09:00 GMT
Hi,

From what I've observed, if you don't provide a script for rack
awareness, hadoop treats all nodes as if they belong the the same rack.

Here is the section in hadoop-site xml where we define our rack
awareness script:

<property>
  <name>topology.script.file.name</name>
  <value>/usr/bin/ip-to-rack.sh</value>
</property>


This is the ip-to-rack.sh script that we use on our cluster.  It assumes
that the reverse lookup of the IP returns a hostname of the form
'compute-x-y', where 'x' is the rack id.  This type of naming is common
for Rocks-managed clusters:

#!/bin/sh

# The default rule assumes that the nodes are connected to the PDU and
switch
# located in the same rack.  Only the exceptions need to be
# explicitly listed here.
for ip in $@ ; do
    hostname=`nslookup $ip | grep "name =" | awk '{print $4}' | sed -e
's/\.local\.$//' `
    case $hostname in
        compute-14-3) rack="/Rack15" ;;
        *)
            rack=`echo $hostname | sed -e
's/^[a-z]*-\([0-9]*\)-[0-9]*.*/\/Rack\1/'`
            ;;
        esac
        echo $rack
done

Hope this helps,

--Mike

On 03/03/2010 05:01 PM, Mag Gam wrote:
> Thanks Alan! Your presentation is very nice!
> 
> "If you don't provide a script for rack awareness, it treats every
> node as if it was its own rack". I am using the default settings and
> the report still says only 1 rack.
> 
> 
> Do you mind sharing a script with us on how you determine a rack? and
> a sample <configuration> </configuration> syntax?
> 
> TIA
> 
> 
> 
> 
> On Wed, Mar 3, 2010 at 11:57 AM, Allen Wittenauer
> <awittenauer@linkedin.com> wrote:
>>
>>
>>
>> On 3/3/10 4:11 AM, "Mag Gam" <magawake@gmail.com> wrote:
>>> An example would be very helpful. There is only 1 paragraph about this
>>> but its far too important not to have an example or two.
>>
>> I covered this in my preso to apachecon last year:
>>
>> http://wiki.apache.org/hadoop/HadoopPresentations?action=AttachFile&do=view&
>> target=aw-apachecon-eu-2009.pdf
>>
>> aka
>>
>> http://bit.ly/d3UU4A
>>
>> You might find the example in/out helpful.  No code, but it is (seriously)
>> trivial to write.
>>
>>
>>



Mime
View raw message