Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 60183 invoked from network); 7 Jul 2010 21:32:24 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 7 Jul 2010 21:32:24 -0000 Received: (qmail 56914 invoked by uid 500); 7 Jul 2010 21:32:21 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 56847 invoked by uid 500); 7 Jul 2010 21:32:21 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 56839 invoked by uid 99); 7 Jul 2010 21:32:20 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 21:32:20 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [129.93.181.2] (HELO mathstat.unl.edu) (129.93.181.2) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 21:32:11 +0000 Received: from ppdm115016.dhcp.fnal.gov (null-00250047ef82.dhcp.fnal.gov [131.225.173.109]) (authenticated bits=0) by mathstat.unl.edu (8.13.8/8.13.8) with ESMTP id o67LVgsN018739 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for ; Wed, 7 Jul 2010 16:31:49 -0500 From: Brian Bockelman Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: multipart/signed; boundary=Apple-Mail-162--416607170; protocol="application/pkcs7-signature"; micalg=sha1 Subject: Re: iozone is not working with HDFS over libfuse Date: Wed, 7 Jul 2010 16:31:43 -0500 In-Reply-To: <201007070956.38710.cray@unix-ag.uni-kl.de> To: common-user@hadoop.apache.org References: <201007061853.19940.cray@unix-ag.uni-kl.de> <4B78E564-A916-4664-8FEE-E9B31DBF68FE@linkedin.com> <201007070956.38710.cray@unix-ag.uni-kl.de> Message-Id: <1AAAA913-C996-4B55-8011-A63D302C5D33@cse.unl.edu> X-Mailer: Apple Mail (2.1081) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-162--416607170 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Jul 7, 2010, at 2:56 AM, Christian Baun wrote: > Hi Brian, >=20 > I wanted to test HDFS against several distributed filesystems. >=20 > Do you know any popular performance benchmarks that run with HDFS?=20 >=20 I can't think of anything off the top of my head. Any ideas out there = on the list? The issue is that HDFS is not very much like a distributed file system. = Maybe a good way is to do one of the MapReduce-based Test IO benchmarks = on top of HDFS versus Lustre versus GPFS, etc? Brian > Best Regards, > Christian=20 >=20 >=20 > Am Dienstag, 6. Juli 2010 schrieb Allen Wittenauer: >>=20 >> On Jul 6, 2010, at 11:56 AM, Brian Bockelman wrote: >>> HDFS does not support the full set of POSIX semantics. I suspect = bonnie++ is exercising things which HDFS doesn't support (such as writes = to a file which is already closed or seeks during writes). >>=20 >>=20 >> In fact, I think the last time this came up a year or two ago, = the stack trace showed that iozone was blowing up on a call to = truncate(), which is definitely not supported. >=20 --Apple-Mail-162--416607170 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIIKDCCA/gw ggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCgmS JomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y aXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAwMDBaFw0xMzAxMjUw ODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEg MB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDEw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYjYaPbCD5mtbiQb7Ka3y1qAm0ZcqKC FciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3KlwjtumTMtOtg35KZCknUd8KM4VGTSFdL VG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0 yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4Vd Gy0eIIPw1pfvYwxO36rm0S109qvbsNlaroPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3 rz9LAgMBAAGjgZ4wgZswDgYDVR0PAQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4E FgQUyhkdEo5upDhdQtQxDgjb2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeow DwYDVR0TAQH/BAUwAwEB/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzAN BgkqhkiG9w0BAQUFAAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLm ph5DBghZUVF8Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4v tQO1KDxgZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3Qghgk FIhmE1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0 gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCgwggMQoAMC AQICAwCvNzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD Ew1ET0VHcmlkcyBDQSAxMB4XDTEwMDUyNDE4MTg0OVoXDTExMDUyNDE4MTg0OVowYTETMBEGCgmS JomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8wDQYDVQQLEwZQZW9wbGUx HzAdBgNVBAMTFkJyaWFuIEJvY2tlbG1hbiA1MDQzMDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQCwiNKRMFRMWG+AJyo/G5bYisPndrH+44JqRHdxDzMaQP59gxrBO2koRwg/13gINe3e J8QL6bX4ANz/y1uglsytmoJwK5J9fxNYgJgbja83kO0j0HNL6oIyBhGStluaNTbiHrk8GA95M9VR XRqpYiwKvT1F0KS2r7sZ+PWevbAek787eTqg51yuvUUlIBPgTm1kV3vZs21oeIZUuw7wPGXBKN49 XqIDsamUIqiFARwPgqKR9eo6itlYy2NrHo0hHLXew37rEOcKv/0g4pI4J/y4+1qB7fN3nMkIMack FWfAQTngcnH/JpKmh8fmXdkeVv8EKYUXIgkUI5pb+Ak105olAgMBAAGjgeAwgd0wEQYJYIZIAYb4 QgEBBAQDAgWgMA4GA1UdDwEB/wQEAwIF4DA2BgNVHSAELzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqG SIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9l Z3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4LmNybDAfBgNVHREEGDAWgRRiYm9ja2VsbUBjc2Uu dW5sLmVkdTAfBgNVHSMEGDAWgBTKGR0Sjm6kOF1C1DEOCNvZjRcNXTANBgkqhkiG9w0BAQUFAAOC AQEAlQSD+8Cvb0GxWqD4xhXd8Sl5MJRr1uJxMeGoMA4RZAJuyvVlBUx8v5moqY0XHMfNI+FulyMx wgOoNfvF3dluz3J4C/u5NvzfNqikLj++sL4XDaZoxSHLo9cJxVTcM15Gogct+kvIF1+msEsqLlNR /lqUVE/o8ANdD6PVx/044f/Dzi6s+6jZmBz/vWPI77ymT1EHaAkaHDqoNIlItPQrAHdkJWY67v1z s6mDrKaspF/2ThDdYax208o1oLFd8wY8kQUdTBlMmUAbchQnjOC9vH17w6meDc8VxD+pEL3vAiG2 JN2vzQ3IJCYTCTmagUyiLWHFEudH8Brn43NY0/HwKTGCAv0wggL5AgEBMHAwaTETMBEGCgmSJomT 8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0 ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAK83MAkGBSsOAwIaBQCgggFi MBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMDcwNzIxMzE0M1ow IwYJKoZIhvcNAQkEMRYEFCT2Kq1w7kDKXu7l5ZDJhaEcY+AUMH8GCSsGAQQBgjcQBDFyMHAwaTET MBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAK83MIGBBgsq hkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERP RUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3Jp ZHMgQ0EgMQIDAK83MA0GCSqGSIb3DQEBAQUABIIBAAJx4kHIEMBhQJbeaxaOo5pZQBEaIDYdwNE4 uqPq+KjyLs+mI1zQsH9zd+O9MDdD27bYBf7ENYB5eqB6tALyTJxQdC7vBW1TxPjjLho0u4rBKiyk soZ4gwpSCClcAdcktP54eIxengsLi39Dw5i1ghNPN6To3emI8GtxYwyZ7PvFHrPJ65boxxT6e6gS cm4chytJNsFRn8ODtdzXd8AczNc90xLPzZZck8DAJ1Nwzaw58AAZmun+8EUWZ+X9oQR9Yynuozzr xwb7nHKkJT9MqffKA2gvsr9rU1/87V4CubIQG3V7evaBDZ3WrWmmAXNUS0BaK31GqU35F73wTplu M8kAAAAAAAA= --Apple-Mail-162--416607170--