Return-Path: Delivered-To: apmail-hadoop-common-dev-archive@www.apache.org Received: (qmail 57287 invoked from network); 28 Apr 2010 13:40:07 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 28 Apr 2010 13:40:07 -0000 Received: (qmail 96472 invoked by uid 500); 28 Apr 2010 13:40:06 -0000 Delivered-To: apmail-hadoop-common-dev-archive@hadoop.apache.org Received: (qmail 96420 invoked by uid 500); 28 Apr 2010 13:40:06 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 96412 invoked by uid 99); 28 Apr 2010 13:40:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 13:40:05 +0000 X-ASF-Spam-Status: No, hits=-1.1 required=10.0 tests=AWL,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [129.93.181.2] (HELO mathstat.unl.edu) (129.93.181.2) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 13:40:00 +0000 Received: from pcp088899pcs.unl.edu (pcp088899pcs.unl.edu [129.93.158.14]) (authenticated bits=0) by mathstat.unl.edu (8.13.8/8.13.8) with ESMTP id o3SDda5R020482 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for ; Wed, 28 Apr 2010 08:39:38 -0500 From: Brian Bockelman Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: multipart/signed; boundary=Apple-Mail-63--50482085; protocol="application/pkcs7-signature"; micalg=sha1 Subject: Re: Regarding Job tracker Date: Wed, 28 Apr 2010 08:39:37 -0500 In-Reply-To: <4BD8083C.7080107@apache.org> To: common-dev@hadoop.apache.org References: <4BD8083C.7080107@apache.org> Message-Id: X-Mailer: Apple Mail (2.1078) --Apple-Mail-63--50482085 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Apr 28, 2010, at 5:04 AM, Steve Loughran wrote: > prajyot bankade wrote: >> Hello Everyone, >> I have just started reading about hadoop job tracker. In one book I = read >> that there is only one job tracker who is responsible to distribute = task to >> worker system. Please make me right if i say some thing wrong. >> I have few questions, >> why there is only one job tracker? >=20 > to provide a single place to make scheduling decisions >=20 (thread hijack) Why is this an advantage? (I mean, I know it's an advantage in terms of = the current architecture... just indulging in some blue-sky thinking = here). One of the projects I work with is the Condor Project out of Madison: http://www.cs.wisc.edu/condor/ who have been building a distributed computing infrastructure for about = 20 years. Here is one of my favorite "overview" papers of theirs: http://www.cs.wisc.edu/condor/doc/condor-practice.pdf (my favorite is = sections 4, 5, and 6.2) They have gotten lots of mileage out of breaking the scheduling and the = resource provision into two different components. Having multiple = jobtrackers would be very advantageous if it didn't require you to = partition your pool. One use case would be to separate out the "production work" from = "research activities". You could have a 'production jobtracker' which = is accessible to a small number of users and has "known good", = pre-approved, business-critical workflows and a 'research jobtracker' = which more folks are allowed to use without pre-approved workflows. = This way, if a researcher accidentally crashes the jobtracker in the = middle of the night, your business-critical work continues. There's plenty of merit to the idea and worth thinking about. Brian --Apple-Mail-63--50482085 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIICjCCA/gw ggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCgmS JomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y aXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAwMDBaFw0xMzAxMjUw ODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEg MB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDEw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYjYaPbCD5mtbiQb7Ka3y1qAm0ZcqKC FciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3KlwjtumTMtOtg35KZCknUd8KM4VGTSFdL VG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0 yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4Vd Gy0eIIPw1pfvYwxO36rm0S109qvbsNlaroPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3 rz9LAgMBAAGjgZ4wgZswDgYDVR0PAQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4E FgQUyhkdEo5upDhdQtQxDgjb2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeow DwYDVR0TAQH/BAUwAwEB/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzAN BgkqhkiG9w0BAQUFAAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLm ph5DBghZUVF8Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4v tQO1KDxgZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3Qghgk FIhmE1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0 gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBAowggLyoAMC AQICAwCB+zANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD Ew1ET0VHcmlkcyBDQSAxMB4XDTA5MDYwMjE5NDExM1oXDTEwMDYwMjE5NDExM1owYTETMBEGCgmS JomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8wDQYDVQQLEwZQZW9wbGUx HzAdBgNVBAMTFkJyaWFuIEJvY2tlbG1hbiA1MDQzMDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDPWEl7hBiuFRVBSY4SwvG0HpkCZi74a0BeD0tNARgxoQVJ7jhJjR3G4y8ino0/5axt 2EEfIWUE+DVpV37IWOQl8q/wdvicnhbfjByxBbq4sfWPLepU7+Kd8k1FKHRHermARn9VxEkFLrLB Gp7O5EX4mFHDaQy+Vv0thtA+m4qKoM+DA/8cOkJA5Rn6ZS/v/vtBzJh9HimVnhBx4+rw2cvKN+7r lKsm7qTn9TCZmrQ97CvBEXSkHS11m8vYF6ZwcTgSCJM0M9nnX5JilupQO1vDICXSUZeWX2xpsqeL x1PFGWgDaYXxFGtTRt2Qc9EPwf9Dr72xGPbKN8u5HylpOMDnAgMBAAGjgcIwgb8wEQYJYIZIAYb4 QgEBBAQDAgWgMA4GA1UdDwEB/wQEAwIF4DAfBgNVHSMEGDAWgBTKGR0Sjm6kOF1C1DEOCNvZjRcN XTAYBgNVHSAEETAPMA0GCyqGSIb3TAMHAQMAMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwu ZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4LmNybDAfBgNVHREEGDAWgRRiYm9ja2VsbUBj c2UudW5sLmVkdTANBgkqhkiG9w0BAQUFAAOCAQEAp6KjcWnfnH/MGlUkUWstE9gtPeymHp+2r4zI w8JXigncJh/8qpSZqBcVhD24WFowI95otblrKYNZKW9f2G/hWwDSxZFqHhCDxFO12vDthrzOc3EH CwypJPvIlZPt/E/x93XruzPxJwPz84DKKuPoJAMeNlADbd+92YtRr2y+VuMpgZaebMAoeCdWH8Cq Y8xheNMajf8uiImBbatDuCu7qRvhwgxsMNLHEt4h853K1Zc181RlFGXG1+uL/Q/8VeKiASiCu+7L 1zpfLg7OCr6rJHb5S7wU+CeAvzSqmyy0fd2mwPeiX7huK+Cw4UjaB3yGKItzWT+KQJnV//wcSrzZ dTGCAv0wggL5AgEBMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERP RUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3Jp ZHMgQ0EgMQIDAIH7MAkGBSsOAwIaBQCgggFiMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJ KoZIhvcNAQkFMQ8XDTEwMDQyODEzMzkzOFowIwYJKoZIhvcNAQkEMRYEFIRVTdbvhC7sMMklkZ01 sMggrFMcMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT 8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE AxMNRE9FR3JpZHMgQ0EgMQIDAIH7MIGBBgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixk ARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBB dXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAIH7MA0GCSqGSIb3DQEBAQUABIIB AGWZTJtV+vMZLZUQXnzwrsZ5j4tczb6gUZKghdkj7NK/JY4e42n6ejRKFNeXV7oI1kxBfRhywQSl LksGqZTy5CrljIGLDxL0k/4UrSMvwTplBZgbSaANPX9UH3VL+JYuSw86CrHYw6dcvSgY1UrFDSWI FIwJOuJu7UIxVXdoNCUrvHc6eR0K9p83yDWc9eyvHNwkq3r0Tr+hEGEWRdhCdYHQQMX607y8gPSg TXvWS+qWszcUCPEK3uzUNKno08jyjctxdqSDQgw+NU5i25Z2nffV6eloztOgKeMS2ggWM1mS1oe8 ciroTfnede8L2/bK7NoV6GFU56mv2XgoGnOZ6msAAAAAAAA= --Apple-Mail-63--50482085--