Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Date: Thu, 6 Oct 2011 08:12:12 -0500
From: Will Maier <wcmaier@hep.wisc.edu>
To: hdfs-user@hadoop.apache.org
Subject: Re: hdfs file - datanode association
Message-ID: <20111006131212.GO5600@burko.lfod.us>
References: <201110061119.20723.vin.boucher@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=sha1; boundary="DEueqSqTbz/jWVG1"
Content-Disposition: inline
In-Reply-To: <201110061119.20723.vin.boucher@gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)


--DEueqSqTbz/jWVG1
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi Vincent-

On Thu, Oct 06, 2011 at 11:19:20AM +0200, Vincent Boucher wrote:
> We are wondering if it is possible to require that the namenode redirects=
 the=20
> blocks of the files of a given directory to a particular set of datanodes?
>=20
> Our case is the following:
>=20
>  - Servers
>     10 x mass storage servers of each 50TB, RAID6
>        -> 500TB available for hdfs
>     30 x working nodes with 6TB (no RAID)
>        -> 180TB available for hdfs
>=20
> We'd like that the files stored in
>    /hdfs/ms
> be hosted on the mass storage (ms) servers,
> while the files in
>   /hdfs/wn
> be hosted on the working nodes (wn).

Why do you want to partition your datanodes this way? In our cluster, datan=
ode
size ranges from ~1 TB (no RAID) up to ~80TB (RAID). While in an ideal worl=
d all
datanodes would be similar in configuration, we have not observed any issues
with this arrangement in production.

--=20

Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/

--DEueqSqTbz/jWVG1
Content-Type: application/x-pkcs7-signature
Content-Disposition: attachment; filename="smime.p7s"
Content-Transfer-Encoding: base64

MIIKzQYJKoZIhvcNAQcCoIIKvjCCCroCAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCC
CCMwggP4MIIC4KADAgECAgEpMA0GCSqGSIb3DQEBBQUAMHUxEzARBgoJkiaJk/IsZAEZFgNu
ZXQxEjAQBgoJkiaJk/IsZAEZFgJFUzEOMAwGA1UEChMFRVNuZXQxIDAeBgNVBAsTF0NlcnRp
ZmljYXRlIEF1dGhvcml0aWVzMRgwFgYDVQQDEw9FU25ldCBSb290IENBIDEwHhcNMDIxMjA1
MDgwMDAwWhcNMTMwMTI1MDgwMDAwWjBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZIm
iZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYw
FAYDVQQDEw1ET0VHcmlkcyBDQSAxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
tPXWI2Gj2wg+ZrW4kG+ymt8tagJtGXKighXIlnH3vCsHLsvbYx7iLC3/WSLXZA1uMcvLLmvZ
yZcdypcI7bpkzLTrYN+SmQpJ1HfCjOFRk0hXS1RvQGzWsnn++nGlQhjNfo8hdC6tN5Grhjj0
3DX2XtU2Wt3tXPa3PJY1eTklmv5n7AgAskucxSFrtMiAzciXX/p9IcGjzed8JZaSd9LWg/iQ
Y0O4dpOMUVs3ms5yzm/Ts7WVUQP9bFc3vonMxYdscGuFXRstHiCD8NaX72MMTt+q5tEtdPar
27DZWq6D0W8XqzympEC6SntN+F3Me4AT6lJPxcaEUVop+RFjd68/SwIDAQABo4GeMIGbMA4G
A1UdDwEB/wQEAwIBhjARBglghkgBhvhCAQEEBAMCAIcwHQYDVR0OBBYEFMoZHRKObqQ4XULU
MQ4I29mNFw1dMB8GA1UdIwQYMBaAFLxdTUgv+DWUWatciUs+0bI6FAHqMA8GA1UdEwEB/wQF
MAMBAf8wJQYDVR0RBB4wHIEaRE9FR3JpZHMtQ0EtMUBkb2Vncmlkcy5vcmcwDQYJKoZIhvcN
AQEFBQADggEBAGTVayAy6nt/QhDoibe0OxKQT4QIoTLw00n/+NrtEjG1JgocMOJi5qYeQwYI
WVFRfGJ+fZEWTJ9UCzrTtR8y6rwLzyE2nGVT5SYBtiF85Uet+mIIWRQMnGyFBDiQwR0eL7UD
tSg8YGcLwG7ZiiKMb4VAqtsbHxbfZAgQZ/iWLUOjuS7ASatz5rrm3jHHO5m0AHyWe4f90IIY
JBSIZhNfiwVzyW5Lj/L4H4OodgSqAH+UxX2RAoBmSopnzdddLQb33nRmcGGHd/xrCXPNwdCT
dYJ8tIKjVT5t/dlHIVxngUax55toJISJK+PDquhJjRlff2ul326ZUFvyVm7T1T1pGoswggQj
MIIDC6ADAgECAgMA1hgwDQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29yZzEY
MBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3Jp
dGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMTAeFw0xMTAzMjExMTM5MzBaFw0xMjAzMjAx
MTM5MzBaMFwxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2Vncmlk
czEPMA0GA1UECxMGUGVvcGxlMRowGAYDVQQDExFXaWxsIE1haWVyIDI4NjMwMjCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBANTfF+rPM01Oi98ATaHgr6qRzoJEEObGjq7XvXNK
pgX9+tFzAUWxAbS2FZjgAmgzB4Qc/tlBW9GcE8xSIoNqK4BD9jcjDZ/RHILKPcVmC78BgO+c
XK73iJVwFEONWC85wZwwTB0a2c+/jXepMRTkF5lsS9B+eTMcoWI2+3CJcaQvURXrrwiLdnbu
Z5vhzPmlRtz/cJyVNQfuP3gZ4g0/52i9cc8ymfOy3TVdjBD8LtlR+by55O3kOwRpN+g8L54o
PD/eGEWvCVlv8UdLrTy8TbIsyhuxEbmzA088P37mWh6Gfc78bVHyMJurLaoOHCzoZr2IWNxf
HYaIxw2sRIgiuEUCAwEAAaOB4DCB3TARBglghkgBhvhCAQEEBAMCBaAwDgYDVR0PAQH/BAQD
AgTwMDYGA1UdIAQvMC0wDQYLKoZIhvdMAwcBAwEwDAYKKoZIhvdMBQICATAOBgwqhkiG90wF
AgMCAQEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL2NybC5kb2Vncmlkcy5vcmcvMWMzZjJj
YTgvMWMzZjJjYTguY3JsMB8GA1UdEQQYMBaBFHdjbWFpZXJAaGVwLndpc2MuZWR1MB8GA1Ud
IwQYMBaAFMoZHRKObqQ4XULUMQ4I29mNFw1dMA0GCSqGSIb3DQEBBQUAA4IBAQCfWXG31UVk
0wWDWWMvJEdUSq1NP6JamSGQRwQBoJ00OdkKY9n+UzevoUiHpTii5kFlxvBUHa+i0hq7wh5k
MwNF5y+8CsRWSD/bRhR30K+2KLyPvpJ2ZbJL9NSVSqVK44ZMquwExzMaktEJlHXY2Y/B51+T
5fWYvHgxzYH7rb1bg1WzFmqlRhimygEOuDCI65ZdaRcBxGU/5UyPJKZmaN3SZH1FAEePfFgl
OHGJCcrIz5lNoGjfENWroeTj8TXU+fbIqvFi64IB/3SOY8fd1PyYI0cGfTOo8VBw2UBokcPP
AkF7t+GQTjAr2a8g+fNr+TEKeWB3I8ShWQRn4/Qsw4xWMYICcjCCAm4CAQEwcDBpMRMwEQYK
CZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0Nl
cnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxAgMA1hgwCQYF
Kw4DAhoFAKCB2DAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0x
MTEwMDYxMzEyMTJaMCMGCSqGSIb3DQEJBDEWBBSJAlJXRsWmFEhQx2IWcevbW1gV+TB5Bgkq
hkiG9w0BCQ8xbDBqMAsGCWCGSAFlAwQBKjALBglghkgBZQMEARYwCwYJYIZIAWUDBAECMAoG
CCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggq
hkiG9w0DAgIBKDANBgkqhkiG9w0BAQEFAASCAQAU5dTqztUNVczVzLV4iyNfp4bROFiYpd/T
2Xu1P1tDd+yt5XlEEnahrtyv3qIIcGMN/fuX/SVFUInt03P7oXFZX4UTRl3TAo4VRkS9oY46
KG/wG09qf2kBz5XcUM8oyfKsV95oSrmgY89TwoHpAvJwfusNPxm9E/JsJW+j7yg1is77nDXh
8X1M7xUi2B430rkhVCfKYgjtVJfyp3Mywmq9bkkzUoHSFUpEu30ooNWufJpbb1YccB68VQrQ
WH2NUG20raR59RhirONvaq0r13MnsbeflrmdgqqEUII7ee0qOznVlaxqjvgSPlY6q6MhfCkQ
nnZJ8NFUgNzPnVPK/Ja1

--DEueqSqTbz/jWVG1--