Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Date: Tue, 11 Oct 2011 12:20:00 -0500
From: Will Maier <wcmaier@hep.wisc.edu>
To: hdfs-user@hadoop.apache.org
Subject: Re: blocks with a huge size?
Message-ID: <20111011172000.GX5600@burko.lfod.us>
References: <201110111039.46437.vin.boucher@gmail.com>
 <20111011104241.GW5600@burko.lfod.us>
 <201110111912.56559.vin.boucher@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=sha1; boundary="KZLWU/9q3evlN4nQ"
Content-Disposition: inline
In-Reply-To: <201110111912.56559.vin.boucher@gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)


--KZLWU/9q3evlN4nQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi Vincent-

On Tue, Oct 11, 2011 at 07:12:55PM +0200, Vincent Boucher wrote:
> Indeed, it's a similar setup, at a smaller scale: Tier2 for CMS at UCLouv=
ain,
> Belgium, with data&sim from the whole collaboration; stageout files from =
the
> Grid for our local users ...

It's always nice to see other CMS sites get involved with HDFS!

[...]
> For us, it is difficult to set a 2x replication while maintaining the cur=
rent
> amount of data we serve to the Collaboration.
>=20
> An alternative would be to switch the mass storage servers with currently=
 one
> large ZFS partition each to servers with as many independent partitions a=
s the
> # of drives they host (typical # of drives per server: 70); and set at the
> Hadoop level a 2x replication. The volume freed by killing the RAID is not
> enough to compensate the replication but that would be a first step.

In fact, we run with exactly this configuration on our larger servers: each=
 disk
is its own RAID0 with a single partition (and 2x replication overall). This
JBOD style is sensitive to disk failure, though the upcoming version is les=
s so.
At our site, we work around the issue by dynamically reconfiguring the
dfs.data.dir parameter in hdfs-site.xml:

    http://hg.hep.wisc.edu/cmsops/hdfs/file/tip/bin/dfs-datadir.sh

Feel free to contact me offlist if you have questions more specific to runn=
ing
HDFS in CMS.

Good luck!

--=20

Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/

--KZLWU/9q3evlN4nQ
Content-Type: application/x-pkcs7-signature
Content-Disposition: attachment; filename="smime.p7s"
Content-Transfer-Encoding: base64

MIIKzQYJKoZIhvcNAQcCoIIKvjCCCroCAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCC
CCMwggP4MIIC4KADAgECAgEpMA0GCSqGSIb3DQEBBQUAMHUxEzARBgoJkiaJk/IsZAEZFgNu
ZXQxEjAQBgoJkiaJk/IsZAEZFgJFUzEOMAwGA1UEChMFRVNuZXQxIDAeBgNVBAsTF0NlcnRp
ZmljYXRlIEF1dGhvcml0aWVzMRgwFgYDVQQDEw9FU25ldCBSb290IENBIDEwHhcNMDIxMjA1
MDgwMDAwWhcNMTMwMTI1MDgwMDAwWjBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZIm
iZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYw
FAYDVQQDEw1ET0VHcmlkcyBDQSAxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
tPXWI2Gj2wg+ZrW4kG+ymt8tagJtGXKighXIlnH3vCsHLsvbYx7iLC3/WSLXZA1uMcvLLmvZ
yZcdypcI7bpkzLTrYN+SmQpJ1HfCjOFRk0hXS1RvQGzWsnn++nGlQhjNfo8hdC6tN5Grhjj0
3DX2XtU2Wt3tXPa3PJY1eTklmv5n7AgAskucxSFrtMiAzciXX/p9IcGjzed8JZaSd9LWg/iQ
Y0O4dpOMUVs3ms5yzm/Ts7WVUQP9bFc3vonMxYdscGuFXRstHiCD8NaX72MMTt+q5tEtdPar
27DZWq6D0W8XqzympEC6SntN+F3Me4AT6lJPxcaEUVop+RFjd68/SwIDAQABo4GeMIGbMA4G
A1UdDwEB/wQEAwIBhjARBglghkgBhvhCAQEEBAMCAIcwHQYDVR0OBBYEFMoZHRKObqQ4XULU
MQ4I29mNFw1dMB8GA1UdIwQYMBaAFLxdTUgv+DWUWatciUs+0bI6FAHqMA8GA1UdEwEB/wQF
MAMBAf8wJQYDVR0RBB4wHIEaRE9FR3JpZHMtQ0EtMUBkb2Vncmlkcy5vcmcwDQYJKoZIhvcN
AQEFBQADggEBAGTVayAy6nt/QhDoibe0OxKQT4QIoTLw00n/+NrtEjG1JgocMOJi5qYeQwYI
WVFRfGJ+fZEWTJ9UCzrTtR8y6rwLzyE2nGVT5SYBtiF85Uet+mIIWRQMnGyFBDiQwR0eL7UD
tSg8YGcLwG7ZiiKMb4VAqtsbHxbfZAgQZ/iWLUOjuS7ASatz5rrm3jHHO5m0AHyWe4f90IIY
JBSIZhNfiwVzyW5Lj/L4H4OodgSqAH+UxX2RAoBmSopnzdddLQb33nRmcGGHd/xrCXPNwdCT
dYJ8tIKjVT5t/dlHIVxngUax55toJISJK+PDquhJjRlff2ul326ZUFvyVm7T1T1pGoswggQj
MIIDC6ADAgECAgMA1hgwDQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29yZzEY
MBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3Jp
dGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMTAeFw0xMTAzMjExMTM5MzBaFw0xMjAzMjAx
MTM5MzBaMFwxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2Vncmlk
czEPMA0GA1UECxMGUGVvcGxlMRowGAYDVQQDExFXaWxsIE1haWVyIDI4NjMwMjCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBANTfF+rPM01Oi98ATaHgr6qRzoJEEObGjq7XvXNK
pgX9+tFzAUWxAbS2FZjgAmgzB4Qc/tlBW9GcE8xSIoNqK4BD9jcjDZ/RHILKPcVmC78BgO+c
XK73iJVwFEONWC85wZwwTB0a2c+/jXepMRTkF5lsS9B+eTMcoWI2+3CJcaQvURXrrwiLdnbu
Z5vhzPmlRtz/cJyVNQfuP3gZ4g0/52i9cc8ymfOy3TVdjBD8LtlR+by55O3kOwRpN+g8L54o
PD/eGEWvCVlv8UdLrTy8TbIsyhuxEbmzA088P37mWh6Gfc78bVHyMJurLaoOHCzoZr2IWNxf
HYaIxw2sRIgiuEUCAwEAAaOB4DCB3TARBglghkgBhvhCAQEEBAMCBaAwDgYDVR0PAQH/BAQD
AgTwMDYGA1UdIAQvMC0wDQYLKoZIhvdMAwcBAwEwDAYKKoZIhvdMBQICATAOBgwqhkiG90wF
AgMCAQEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL2NybC5kb2Vncmlkcy5vcmcvMWMzZjJj
YTgvMWMzZjJjYTguY3JsMB8GA1UdEQQYMBaBFHdjbWFpZXJAaGVwLndpc2MuZWR1MB8GA1Ud
IwQYMBaAFMoZHRKObqQ4XULUMQ4I29mNFw1dMA0GCSqGSIb3DQEBBQUAA4IBAQCfWXG31UVk
0wWDWWMvJEdUSq1NP6JamSGQRwQBoJ00OdkKY9n+UzevoUiHpTii5kFlxvBUHa+i0hq7wh5k
MwNF5y+8CsRWSD/bRhR30K+2KLyPvpJ2ZbJL9NSVSqVK44ZMquwExzMaktEJlHXY2Y/B51+T
5fWYvHgxzYH7rb1bg1WzFmqlRhimygEOuDCI65ZdaRcBxGU/5UyPJKZmaN3SZH1FAEePfFgl
OHGJCcrIz5lNoGjfENWroeTj8TXU+fbIqvFi64IB/3SOY8fd1PyYI0cGfTOo8VBw2UBokcPP
AkF7t+GQTjAr2a8g+fNr+TEKeWB3I8ShWQRn4/Qsw4xWMYICcjCCAm4CAQEwcDBpMRMwEQYK
CZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0Nl
cnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxAgMA1hgwCQYF
Kw4DAhoFAKCB2DAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0x
MTEwMTExNzIwMDBaMCMGCSqGSIb3DQEJBDEWBBQCTd9pHtq3OwXzBtTEIGXSd3SD+zB5Bgkq
hkiG9w0BCQ8xbDBqMAsGCWCGSAFlAwQBKjALBglghkgBZQMEARYwCwYJYIZIAWUDBAECMAoG
CCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggq
hkiG9w0DAgIBKDANBgkqhkiG9w0BAQEFAASCAQCDJDPGnoLL35Uw3Rw2EsPA/Bn4+yk0kmRM
Rwodz9qKd8t5oWZRKuFr7CnFMjrpH5dYVqEztsyXk6YSsaCjK9MXrTWbMSMQyn4nh+nWiWzj
RwSoaaBZT8lfSi3ZshOtZy3uAraonnsIl5F2wYLsHGRKzvUuvUoM+ivL0J0sNL1xHId0dfk9
E72WvVg10SZ8T2XyZqozphUMa/QzUIY1uTzVXKInEkHdawRX6KRUBiFZZ2X/dTCLGWGWL7dt
WpBapTWP71MtnPSQrWV2PF14ZcIwOQScMZ3SarKob8MQhts51wfYbQtVE3NAZhitJKGpNu4u
mxUiJQAnB6Y9Gh8J3Y8S

--KZLWU/9q3evlN4nQ--