Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 45062 invoked from network); 1 Sep 2009 12:00:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 1 Sep 2009 12:00:14 -0000 Received: (qmail 1920 invoked by uid 500); 1 Sep 2009 12:00:12 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 1824 invoked by uid 500); 1 Sep 2009 12:00:12 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 1814 invoked by uid 99); 1 Sep 2009 12:00:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 12:00:12 +0000 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [129.93.181.2] (HELO mathstat.unl.edu) (129.93.181.2) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 12:00:02 +0000 Received: from [192.168.0.102] (user-0cdvqhn.cable.mindspring.com [24.223.234.55]) (authenticated bits=0) by mathstat.unl.edu (8.13.8/8.13.8) with ESMTP id n81BxcMk010809 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for ; Tue, 1 Sep 2009 06:59:41 -0500 Message-Id: <1FB3CB82-A660-4DA2-8567-08CC3CFA1A23@cse.unl.edu> From: Brian Bockelman To: common-user@hadoop.apache.org In-Reply-To: <944750dd0909010208q31c33e5bwde3fc2d1ca1e5bdd@mail.gmail.com> Content-Type: multipart/signed; boundary=Apple-Mail-85-768755074; micalg=sha1; protocol="application/pkcs7-signature" Mime-Version: 1.0 (Apple Message framework v936) Subject: Re: Datanode high memory usage Date: Tue, 1 Sep 2009 06:59:38 -0500 References: <77938bc20908310340g63c6b29aw25a4edd16f1fc840@mail.gmail.com> <007e01ca2abd$d659b7f0$830d27d0$@com> <77938bc20908310653t13ef7031i25487d7c818fef8b@mail.gmail.com> <8211a1320908310840p59e56e16o2e933a151a951334@mail.gmail.com> <77938bc20908310907g542e50d2me9a6f31e8d4ba3d4@mail.gmail.com> <7a8854060908311004ncd63e7cv9ea388a3e86d25f1@mail.gmail.com> <77938bc20908311009n3e6a6ebg7c268d2a5ab46530@mail.gmail.com> <944750dd0908311840p8062527l350180cfa451f7ec@mail.gmail.com> <77938bc20909010159lb667aecp12417c6f6da06c3f@mail.gmail.com> <944750dd0909010208q31c33e5bwde3fc2d1ca1e5bdd@mail.gmail.com> X-Mailer: Apple Mail (2.936) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-85-768755074 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Hey Mafish, If you are getting 1-2m blocks on a single datanode, you'll have many other problems - especially with regards to periodic block reports. With regards to memory, have you tried the compressed pointers JDK option (we saw great benefits on the NN)? Java is incredibly hard to get a straight answer from with regards to memory. You need to perform a GC first manually - the actual usage is the amount it reports used post-GC. You can get these details by using JMX. Brian On Sep 1, 2009, at 4:08 AM, Mafish Liu wrote: > Both NameNode and DataNode will be affected by number of files > greatly. > In my test, almost 60% memory are used in datanodes while storing 1m > files, and the value reach 80% with 2m files. > My test best is with 5 nodes, 1 namenode and 4 datanodes. All nodes > have 2GB memory and replication is 3. > > 2009/9/1 Stas Oskin : >> Hi. >> >> 2009/9/1 Mafish Liu >> >>> Did you have many small files in your system? >>> >>> >> Yes, quite plenty. >> >> But this should influence the Namenode, and not the Datanode, >> correct? >> >> Regards. >> > > > > -- > Mafish@gmail.com --Apple-Mail-85-768755074 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIICjCCA/gw ggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCgmS JomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y aXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAwMDBaFw0xMzAxMjUw ODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEg MB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDEw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYjYaPbCD5mtbiQb7Ka3y1qAm0ZcqKC FciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3KlwjtumTMtOtg35KZCknUd8KM4VGTSFdL VG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0 yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4Vd Gy0eIIPw1pfvYwxO36rm0S109qvbsNlaroPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3 rz9LAgMBAAGjgZ4wgZswDgYDVR0PAQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4E FgQUyhkdEo5upDhdQtQxDgjb2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeow DwYDVR0TAQH/BAUwAwEB/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzAN BgkqhkiG9w0BAQUFAAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLm ph5DBghZUVF8Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4v tQO1KDxgZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3Qghgk FIhmE1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0 gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBAowggLyoAMC AQICAwCB+zANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD Ew1ET0VHcmlkcyBDQSAxMB4XDTA5MDYwMjE5NDExM1oXDTEwMDYwMjE5NDExM1owYTETMBEGCgmS JomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8wDQYDVQQLEwZQZW9wbGUx HzAdBgNVBAMTFkJyaWFuIEJvY2tlbG1hbiA1MDQzMDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDPWEl7hBiuFRVBSY4SwvG0HpkCZi74a0BeD0tNARgxoQVJ7jhJjR3G4y8ino0/5axt 2EEfIWUE+DVpV37IWOQl8q/wdvicnhbfjByxBbq4sfWPLepU7+Kd8k1FKHRHermARn9VxEkFLrLB Gp7O5EX4mFHDaQy+Vv0thtA+m4qKoM+DA/8cOkJA5Rn6ZS/v/vtBzJh9HimVnhBx4+rw2cvKN+7r lKsm7qTn9TCZmrQ97CvBEXSkHS11m8vYF6ZwcTgSCJM0M9nnX5JilupQO1vDICXSUZeWX2xpsqeL x1PFGWgDaYXxFGtTRt2Qc9EPwf9Dr72xGPbKN8u5HylpOMDnAgMBAAGjgcIwgb8wEQYJYIZIAYb4 QgEBBAQDAgWgMA4GA1UdDwEB/wQEAwIF4DAfBgNVHSMEGDAWgBTKGR0Sjm6kOF1C1DEOCNvZjRcN XTAYBgNVHSAEETAPMA0GCyqGSIb3TAMHAQMAMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwu ZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4LmNybDAfBgNVHREEGDAWgRRiYm9ja2VsbUBj c2UudW5sLmVkdTANBgkqhkiG9w0BAQUFAAOCAQEAp6KjcWnfnH/MGlUkUWstE9gtPeymHp+2r4zI w8JXigncJh/8qpSZqBcVhD24WFowI95otblrKYNZKW9f2G/hWwDSxZFqHhCDxFO12vDthrzOc3EH CwypJPvIlZPt/E/x93XruzPxJwPz84DKKuPoJAMeNlADbd+92YtRr2y+VuMpgZaebMAoeCdWH8Cq Y8xheNMajf8uiImBbatDuCu7qRvhwgxsMNLHEt4h853K1Zc181RlFGXG1+uL/Q/8VeKiASiCu+7L 1zpfLg7OCr6rJHb5S7wU+CeAvzSqmyy0fd2mwPeiX7huK+Cw4UjaB3yGKItzWT+KQJnV//wcSrzZ dTGCAv0wggL5AgEBMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERP RUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3Jp ZHMgQ0EgMQIDAIH7MAkGBSsOAwIaBQCgggFiMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJ KoZIhvcNAQkFMQ8XDTA5MDkwMTExNTkzOFowIwYJKoZIhvcNAQkEMRYEFA9kHwLpWlg0B/UKA5Md ltjcDdabMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT 8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE AxMNRE9FR3JpZHMgQ0EgMQIDAIH7MIGBBgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixk ARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBB dXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAIH7MA0GCSqGSIb3DQEBAQUABIIB AHbU8tO8w6etOn3xFHVavq0AaCRBpxX55uX7Tzte9qbtypwTaZSRANeQxOiBvt1Phv7xfBkjPk0u yJHcpPkKERjvCanLS/+rzeu04+U7jShoaImlhHgkMeZ/9Inmz1vMDNUYy7IUbR+Wz0lJoct+pevH cFC8mNIt3aoTAv+NxLaHNzw+jg1Ud8K9OXkQtyB7U8FB5CvZQkfMRvf7NYSmBHgCWVeqH26E/GLO SVSTj/svDk05c669ASUcPd8S5HxShWHVS5OQRkHOamDltA3grqF8JpCLs6d3MHmMHm2vDanUkFY1 cCjmFfHlwMLzsuZJ8HCCeTYxCpvP6yiRwChxapMAAAAAAAA= --Apple-Mail-85-768755074--