hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Thomas <tho...@hep.caltech.edu>
Subject Re: rack awareness help
Date Thu, 04 Mar 2010 01:09:00 GMT

From what I've observed, if you don't provide a script for rack
awareness, hadoop treats all nodes as if they belong the the same rack.

Here is the section in hadoop-site xml where we define our rack
awareness script:


This is the ip-to-rack.sh script that we use on our cluster.  It assumes
that the reverse lookup of the IP returns a hostname of the form
'compute-x-y', where 'x' is the rack id.  This type of naming is common
for Rocks-managed clusters:


# The default rule assumes that the nodes are connected to the PDU and
# located in the same rack.  Only the exceptions need to be
# explicitly listed here.
for ip in $@ ; do
    hostname=`nslookup $ip | grep "name =" | awk '{print $4}' | sed -e
's/\.local\.$//' `
    case $hostname in
        compute-14-3) rack="/Rack15" ;;
            rack=`echo $hostname | sed -e
        echo $rack

Hope this helps,


On 03/03/2010 05:01 PM, Mag Gam wrote:
> Thanks Alan! Your presentation is very nice!
> "If you don't provide a script for rack awareness, it treats every
> node as if it was its own rack". I am using the default settings and
> the report still says only 1 rack.
> Do you mind sharing a script with us on how you determine a rack? and
> a sample <configuration> </configuration> syntax?
> On Wed, Mar 3, 2010 at 11:57 AM, Allen Wittenauer
> <awittenauer@linkedin.com> wrote:
>> On 3/3/10 4:11 AM, "Mag Gam" <magawake@gmail.com> wrote:
>>> An example would be very helpful. There is only 1 paragraph about this
>>> but its far too important not to have an example or two.
>> I covered this in my preso to apachecon last year:
>> http://wiki.apache.org/hadoop/HadoopPresentations?action=AttachFile&do=view&
>> target=aw-apachecon-eu-2009.pdf
>> aka
>> http://bit.ly/d3UU4A
>> You might find the example in/out helpful.  No code, but it is (seriously)
>> trivial to write.

View raw message