Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 4956 invoked from network); 7 Sep 2009 19:09:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 7 Sep 2009 19:09:15 -0000 Received: (qmail 48874 invoked by uid 500); 7 Sep 2009 19:09:12 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 48792 invoked by uid 500); 7 Sep 2009 19:09:12 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 48781 invoked by uid 99); 7 Sep 2009 19:09:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Sep 2009 19:09:12 +0000 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [129.93.181.2] (HELO mathstat.unl.edu) (129.93.181.2) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Sep 2009 19:09:03 +0000 Received: from [192.168.0.102] (user-0cdvqhn.cable.mindspring.com [24.223.234.55]) (authenticated bits=0) by mathstat.unl.edu (8.13.8/8.13.8) with ESMTP id n87J8dpP005530 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for ; Mon, 7 Sep 2009 14:08:42 -0500 From: Brian Bockelman Mime-Version: 1.0 (Apple Message framework v1075.2) Content-Type: multipart/signed; boundary=Apple-Mail-2--834587751; protocol="application/pkcs7-signature"; micalg=sha1 Subject: Re: HDFS and Linux File System Date: Mon, 7 Sep 2009 14:08:39 -0500 In-Reply-To: <17e273100909070218v4659e2dcu8598207843adeed6@mail.gmail.com> To: common-user@hadoop.apache.org References: <17e273100909070218v4659e2dcu8598207843adeed6@mail.gmail.com> Message-Id: <2EBE5668-CC80-4151-8577-2A923C4A1AFF@cse.unl.edu> X-Mailer: Apple Mail (2.1075.2) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-2--834587751 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Hey Ted, It's hard to avoid copying files, unless if you are able to change your application to talk to HDFS directly (and even then, there are a lot of "gotchas" that you wouldn't have to put up with at an application level -- look at the Chukwa paper). I would advise looking at Chukwa, http://wiki.apache.org/hadoop/ Chukwa, and then rotating logfiles quickly. Facebook's Scribe is supposed to do this sort of stuff too (and is very impressive), but I'm not familiar with it. On face value, it appears that it might take more effort to get scribe well-integrated, but it would have more functionality. Brian On Sep 7, 2009, at 4:18 AM, Ted Yu wrote: > We're using hadoop 0.20.0 to analyze large log files from web servers. > I am looking for better HDFS support so that I don't have to copy > log files > from Linux File System over. > > Please comment. > > Thanks --Apple-Mail-2--834587751 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIICjCCA/gw ggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCgmS JomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y aXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAwMDBaFw0xMzAxMjUw ODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEg MB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDEw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYjYaPbCD5mtbiQb7Ka3y1qAm0ZcqKC FciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3KlwjtumTMtOtg35KZCknUd8KM4VGTSFdL VG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0 yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4Vd Gy0eIIPw1pfvYwxO36rm0S109qvbsNlaroPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3 rz9LAgMBAAGjgZ4wgZswDgYDVR0PAQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4E FgQUyhkdEo5upDhdQtQxDgjb2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeow DwYDVR0TAQH/BAUwAwEB/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzAN BgkqhkiG9w0BAQUFAAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLm ph5DBghZUVF8Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4v tQO1KDxgZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3Qghgk FIhmE1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0 gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBAowggLyoAMC AQICAwCB+zANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD Ew1ET0VHcmlkcyBDQSAxMB4XDTA5MDYwMjE5NDExM1oXDTEwMDYwMjE5NDExM1owYTETMBEGCgmS JomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8wDQYDVQQLEwZQZW9wbGUx HzAdBgNVBAMTFkJyaWFuIEJvY2tlbG1hbiA1MDQzMDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDPWEl7hBiuFRVBSY4SwvG0HpkCZi74a0BeD0tNARgxoQVJ7jhJjR3G4y8ino0/5axt 2EEfIWUE+DVpV37IWOQl8q/wdvicnhbfjByxBbq4sfWPLepU7+Kd8k1FKHRHermARn9VxEkFLrLB Gp7O5EX4mFHDaQy+Vv0thtA+m4qKoM+DA/8cOkJA5Rn6ZS/v/vtBzJh9HimVnhBx4+rw2cvKN+7r lKsm7qTn9TCZmrQ97CvBEXSkHS11m8vYF6ZwcTgSCJM0M9nnX5JilupQO1vDICXSUZeWX2xpsqeL x1PFGWgDaYXxFGtTRt2Qc9EPwf9Dr72xGPbKN8u5HylpOMDnAgMBAAGjgcIwgb8wEQYJYIZIAYb4 QgEBBAQDAgWgMA4GA1UdDwEB/wQEAwIF4DAfBgNVHSMEGDAWgBTKGR0Sjm6kOF1C1DEOCNvZjRcN XTAYBgNVHSAEETAPMA0GCyqGSIb3TAMHAQMAMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwu ZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4LmNybDAfBgNVHREEGDAWgRRiYm9ja2VsbUBj c2UudW5sLmVkdTANBgkqhkiG9w0BAQUFAAOCAQEAp6KjcWnfnH/MGlUkUWstE9gtPeymHp+2r4zI w8JXigncJh/8qpSZqBcVhD24WFowI95otblrKYNZKW9f2G/hWwDSxZFqHhCDxFO12vDthrzOc3EH CwypJPvIlZPt/E/x93XruzPxJwPz84DKKuPoJAMeNlADbd+92YtRr2y+VuMpgZaebMAoeCdWH8Cq Y8xheNMajf8uiImBbatDuCu7qRvhwgxsMNLHEt4h853K1Zc181RlFGXG1+uL/Q/8VeKiASiCu+7L 1zpfLg7OCr6rJHb5S7wU+CeAvzSqmyy0fd2mwPeiX7huK+Cw4UjaB3yGKItzWT+KQJnV//wcSrzZ dTGCAv0wggL5AgEBMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERP RUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3Jp ZHMgQ0EgMQIDAIH7MAkGBSsOAwIaBQCgggFiMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJ KoZIhvcNAQkFMQ8XDTA5MDkwNzE5MDgzOVowIwYJKoZIhvcNAQkEMRYEFPqzUny5Ge6UkzxS+FsS eH29X/KIMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT 8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE AxMNRE9FR3JpZHMgQ0EgMQIDAIH7MIGBBgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixk ARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBB dXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAIH7MA0GCSqGSIb3DQEBAQUABIIB AIE5/a1t2kqCLGGGqUTyHmPWnEyV/PhyK/6vmy2lfnJVzZOTaeZUCPftI+sfKg9p3g+SUM0/idF9 HwEXeq1ts4BghvmakyLuD2lGcFcVHDFyNLuBidvIJnsma+MUn3Zntb4S9Jq0L6ngx5JSMWeSnYWn T4UzDOfj5wNNAxBb+adKX/o4ZZquzUf+T0d8+mKY1Z1GG6d1OYXGrXdB3Bsk/62yqsFRs31EKE8m bxzokJGNY1jyJ1badZxgtSOupiHWihd8Hq0Ag8kKnwCr+CF6sTDjPELW++3+W/g6AxQ0v92DNE/G o0kZtCO5HLeKwwfdRr7dSSYdT0VmuBVh5dxqSGUAAAAAAAA= --Apple-Mail-2--834587751--