Return-Path: 
 <general-return-1216-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71884 invoked from network); 19 Mar 2010 01:11:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 01:11:51 -0000
Received: (qmail 882 invoked by uid 500); 19 Mar 2010 01:11:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 838 invoked by uid 500); 19 Mar 2010 01:11:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 829 invoked by uid 99); 19 Mar 2010 01:11:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 01:11:50 +0000
X-ASF-Spam-Status: No, hits=-1.0 required=10.0
	tests=AWL,SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [64.81.36.244] (HELO foobar.kobold.org) (64.81.36.244)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 01:11:45 +0000
Received: from owl.kobold.org (owl.kobold.org [192.168.1.7])
	(authenticated bits=0)
	by foobar.kobold.org (8.14.1/8.14.1) with ESMTP id o2INvO3q008079
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 18 Mar 2010 16:57:30 -0700
Message-ID: <4BA2CF33.1000504@hep.caltech.edu>
Date: Thu, 18 Mar 2010 18:11:15 -0700
From: Michael Thomas <thomas@hep.caltech.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.1.7) Gecko/20100120 Fedora/3.0.1-1.fc12 Thunderbird/3.0.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: rack awareness help
References: <C7B48351.36DF%awittenauer@linkedin.com>
	 <4B8FC1CC.2090108@apache.org>
	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>
	 <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
	 <9db00c201003180015n1e4e8941i2c53296edd21258f@mail.gmail.com>
 <1cbd6f831003181741t2f538eb8xc872cba6dfe00761@mail.gmail.com>
In-Reply-To: <1cbd6f831003181741t2f538eb8xc872cba6dfe00761@mail.gmail.com>
Content-Type: multipart/signed; protocol="application/pkcs7-signature";
 micalg=sha1; boundary="------------ms070408020806080508010302"

--------------ms070408020806080508010302
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

On 03/18/2010 05:41 PM, Mag Gam wrote:
> Chris:
>
> This clears up my questions a lot! Thankyou.
>
> So, if I have 4 data servers and I want 2 racks. I can do this
>
> #!/bin/bash
> #rack1.sh
> echo rack1
>
> #bin/bash
> #rack2.sh
> echo rack2
>
>
> So, I can do this for 2 servers
>
>
> <property>
>   <name>topology.script.file.name</name>
>   <value>rack1.sh</value>
> </property>
>
> And for the other 2 servers, I can do this:
>
>
> <property>
>   <name>topology.script.file.name</name>
>   <value>rack2.sh</value>
> </property>
>
>
> correct?

Incorrect.  You only specify a single topology.script.file.name on the=20
namenode.  This attribute is ignored on the datanodes.

Also be aware that the namenode invokes the script with the _ip address_ =

of each datanode, not the hostname of each datanode.  If you need to=20
convert the ip address to a hostname in order to determine which rack=20
the datanode is on, then you have to put that conversion logic in your=20
topology.script.file.name.

--Mike

> On Thu, Mar 18, 2010 at 3:15 AM, Christopher Tubbs<ctubbsii@gmail.com> =
 wrote:
>> Hadoop will identify data nodes in your cluster by name and execute
>> your script with the data node as an argument. The expected output of
>> your script is the name of the rack on which it is located.
>>
>> The script you referenced takes the node name as an argument ($1), and=

>> crawls through a separate file looking up that node in the left
>> column, and printing the value in the second column if it finds it.
>>
>> If you were to use this script, you would just create the topology
>> file that lists all your nodes by name/ip on the left and the rack
>> they are in on the right.
>>
>> On Wed, Mar 17, 2010 at 11:34 PM, Mag Gam<magawake@gmail.com>  wrote:
>>> Well,  I didn't really solve the problem. Now I have even more questi=
ons.
>>>
>>> I came across this script,
>>> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>>>
>>> but it makes no sense to me! Can someone please try to explain what
>>> its trying to do?
>>>
>>>
>>> MikeThomas:
>>>
>>> Your script isn't working for me. I think there are some syntax
>>> errors. Is this how its supposed to look: http://pastebin.ca/1844287
>>>
>>> thanks
>>>
>>>
>>>
>>> On Thu, Mar 4, 2010 at 10:30 PM, Jeff Hammerbacher<hammer@cloudera.co=
m>  wrote:
>>>> Hey Mag,
>>>>
>>>> Glad you have solved the problem. I've created a JIRA ticket to impr=
ove the
>>>> existing documentation: https://issues.apache.org/jira/browse/HADOOP=
-6616.
>>>> If you have some time, it would be useful to hear what could be adde=
d to the
>>>> existing documentation that would have helped you figure this out so=
oner.
>>>>
>>>> Thanks,
>>>> Jeff
>>>>
>>>> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam<magawake@gmail.com>  wrote:
>>>>
>>>>> Thanks everyone for explaining this to me instead of giving me RTFM=
!
>>>>>
>>>>> I will play around with it and see how far I get.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran<stevel@apache.org>  =
wrote:
>>>>>> Allen Wittenauer wrote:
>>>>>>>
>>>>>>> On 3/3/10 5:01 PM, "Mag Gam"<magawake@gmail.com>  wrote:
>>>>>>>
>>>>>>>> Thanks Alan! Your presentation is very nice!
>>>>>>>
>>>>>>> Thanks. :)
>>>>>>>
>>>>>>>> "If you don't provide a script for rack awareness, it treats eve=
ry
>>>>>>>> node as if it was its own rack". I am using the default settings=
 and
>>>>>>>> the report still says only 1 rack.
>>>>>>>
>>>>>>> Let's take a different approach to convince you. :)
>>>>>>>
>>>>>>> Think about the question:  Is there a difference between all node=
s in
>>>>> one
>>>>>>> rack vs. every node acting as a lone rack?
>>>>>>>
>>>>>>> The answer is no, there isn't any difference.  In both cases, all=
 copies
>>>>>>> of
>>>>>>> the blocks can go to pretty much any node. When a MR job runs, ev=
ery
>>>>> node
>>>>>>> would either be considered 'off rack' or 'rack-local'.
>>>>>>>
>>>>>>> So there is no difference.
>>>>>>>
>>>>>>>
>>>>>>>> Do you mind sharing a script with us on how you determine a rack=
? and
>>>>>>>> a sample<configuration>  </configuration>  syntax?
>>>>>>>
>>>>>>> Michael has already posted his, so I'll skip this one. :)
>>>>>>>
>>>>>>
>>>>>> Think Mag probably wanted a shell script.
>>>>>>
>>>>>> Mag, give your machines IPv4 addresses that map to rack number. 10=
=2E1.1.*
>>>>> for
>>>>>> rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP ad=
dress
>>>>> by
>>>>>> the top bytes, returning "10.1.1" for everything in rack one, "10.=
1.2"
>>>>> for
>>>>>> rack 2; Hadoop will be happy
>>>>>>
>>>>>
>>>>
>>>
>>



--------------ms070408020806080508010302
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMVjCC
A/gwggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDES
MBAGCgmSJomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNh
dGUgQXV0aG9yaXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAw
MDBaFw0xMzAxMjUwODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/Is
ZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNV
BAMTDURPRUdyaWRzIENBIDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYj
YaPbCD5mtbiQb7Ka3y1qAm0ZcqKCFciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3K
lwjtumTMtOtg35KZCknUd8KM4VGTSFdLVG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe
1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2
k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4VdGy0eIIPw1pfvYwxO36rm0S109qvbsNla
roPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3rz9LAgMBAAGjgZ4wgZswDgYDVR0P
AQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4EFgQUyhkdEo5upDhdQtQxDgjb
2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeowDwYDVR0TAQH/BAUwAwEB
/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzANBgkqhkiG9w0BAQUF
AAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLmph5DBghZUVF8
Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4vtQO1KDxg
ZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3QghgkFIhm
E1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCkwggMR
oAMCAQICAwChZzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYK
CZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVz
MRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxMB4XDTEwMDEyNTE4MzEyMFoXDTExMDEyNTE4MzEy
MFowYDETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8w
DQYDVQQLEwZQZW9wbGUxHjAcBgNVBAMTFU1pY2hhZWwgVGhvbWFzIDcyNTM3NDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBALchCLXStiJKw5u2+nFW0020jqo1X8/kRB9kus8a
6Op5ds3j5O3IZkyZ41tQwE9Rc0yRDiDGpT1jL/n2LK1HPXvZbasmPCxxTMY3kNTm4fWAnRS7
UzLo0Adaz81bcT8Buo7Il1FCS684LhsK19JChqU2/iY8G7H0uBImj6QDLR0+w5QsLfD7aOXx
M8Njhse1ZqMyjwjhw5FqWyWNhlKi1oW42eXtYhbsqyAQZSXLoz2SwqBnLTRqcRMn4GgvB83Z
1FObyb79HFiuqlbaMlXWy0s8ywajJnMG8LsgwkNJZQLJsPalz5rwe/pOG1abasLMBVpJGGSB
AQktY8s5M+U9RlMCAwEAAaOB4jCB3zARBglghkgBhvhCAQEEBAMCBeAwDgYDVR0PAQH/BAQD
AgTwMDYGA1UdIAQvMC0wDQYLKoZIhvdMAwcBAwEwDAYKKoZIhvdMBQICATAOBgwqhkiG90wF
AgMCAQEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL2NybC5kb2Vncmlkcy5vcmcvMWMzZjJj
YTgvMWMzZjJjYTguY3JsMCEGA1UdEQQaMBiBFnRob21hc0BoZXAuY2FsdGVjaC5lZHUwHwYD
VR0jBBgwFoAUyhkdEo5upDhdQtQxDgjb2Y0XDV0wDQYJKoZIhvcNAQEFBQADggEBAGcleFpm
d9Ho37gHZrrF0zulSQ3Aemrbx2O4EVJotQbtAs2v4nnH0eAa0F37BXRA6DAvgaC4YODA6z5M
HHE2mnsIebv0nS8DLj1yN7C1nqDSUXlaIzExEGQ1vXG2XlVo6pMZgJd2ML9+dyUUOcnNj0kQ
GeMysWBHbDChdfVG8y2PuGzoFY/ikNtWWhTNcPVZHIL5I2JCYqV5lyLQhlkC0Q5eqbqYKAq1
GuQmLO8hPfzkb6qbAkzxB8/FFqjlPgBgFcd9Nf2fAI14kI77iRbEuu8jSTPx13p6uYvcX4hy
1cW4Y7lnE8bBkAFWrUuNGc+aw7Ang6rJANWXtF/tN+UqUOcwggQpMIIDEaADAgECAgMAoWcw
DQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkW
CERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMN
RE9FR3JpZHMgQ0EgMTAeFw0xMDAxMjUxODMxMjBaFw0xMTAxMjUxODMxMjBaMGAxEzARBgoJ
kiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2VncmlkczEPMA0GA1UECxMGUGVv
cGxlMR4wHAYDVQQDExVNaWNoYWVsIFRob21hcyA3MjUzNzQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC3IQi10rYiSsObtvpxVtNNtI6qNV/P5EQfZLrPGujqeXbN4+TtyGZM
meNbUMBPUXNMkQ4gxqU9Yy/59iytRz172W2rJjwscUzGN5DU5uH1gJ0Uu1My6NAHWs/NW3E/
AbqOyJdRQkuvOC4bCtfSQoalNv4mPBux9LgSJo+kAy0dPsOULC3w+2jl8TPDY4bHtWajMo8I
4cORalsljYZSotaFuNnl7WIW7KsgEGUly6M9ksKgZy00anETJ+BoLwfN2dRTm8m+/RxYrqpW
2jJV1stLPMsGoyZzBvC7IMJDSWUCybD2pc+a8Hv6ThtWm2rCzAVaSRhkgQEJLWPLOTPlPUZT
AgMBAAGjgeIwgd8wEQYJYIZIAYb4QgEBBAQDAgXgMA4GA1UdDwEB/wQEAwIE8DA2BgNVHSAE
LzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqGSIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1Ud
HwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4
LmNybDAhBgNVHREEGjAYgRZ0aG9tYXNAaGVwLmNhbHRlY2guZWR1MB8GA1UdIwQYMBaAFMoZ
HRKObqQ4XULUMQ4I29mNFw1dMA0GCSqGSIb3DQEBBQUAA4IBAQBnJXhaZnfR6N+4B2a6xdM7
pUkNwHpq28djuBFSaLUG7QLNr+J5x9HgGtBd+wV0QOgwL4GguGDgwOs+TBxxNpp7CHm79J0v
Ay49cjewtZ6g0lF5WiMxMRBkNb1xtl5VaOqTGYCXdjC/fnclFDnJzY9JEBnjMrFgR2wwoXX1
RvMtj7hs6BWP4pDbVloUzXD1WRyC+SNiQmKleZci0IZZAtEOXqm6mCgKtRrkJizvIT385G+q
mwJM8QfPxRao5T4AYBXHfTX9nwCNeJCO+4kWxLrvI0kz8dd6ermL3F+IctXFuGO5ZxPGwZAB
Vq1LjRnPmsOwJ4OqyQDVl7Rf7TflKlDnMYIDXjCCA1oCAQEwcDBpMRMwEQYKCZImiZPyLGQB
GRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRl
IEF1dGhvcml0aWVzMRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxAgMAoWcwCQYFKw4DAhoFAKCC
AcMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTAwMzE5MDEx
MTE1WjAjBgkqhkiG9w0BCQQxFgQUDWVt/RDszyoVr1CGTYonVGZ4M5cwXwYJKoZIhvcNAQkP
MVIwUDALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3
DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEG
CgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAKFnMIGB
BgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixk
ARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE
AxMNRE9FR3JpZHMgQ0EgMQIDAKFnMA0GCSqGSIb3DQEBAQUABIIBAIsgUf2uvuEeyHHL5IYo
4AUkMk2q87mWy9z8wtjg3bQ1o43XbRefwtUzWJJeRCIB67HTyhk7PzLwyeu4DPKkoBV6GgSQ
6yCYOANf9bWRMeavCAcFkZIDnOO2YSKrrLZUrGpdx5qI+DVg9H1MXHcotQ4109gZxEpKnCM0
/qECpLMQTCZtoieYAXr5xN0V4IdqyMm65AOSIEdm7pRqeuK5aqa27RCZMV2D2zkGfXfTyuU3
qZI7SSH1LsGDcPuV2Gz4KKbRvdPHkAypfevHfMrvwafcaYqoq3NNW0f8fO08D7KxBUTebNZl
eheDQDnJdyaTFkwevdp9HGNHrkAGnT0lef8AAAAAAAA=
--------------ms070408020806080508010302--