From hdfs-dev-return-994-apmail-hadoop-hdfs-dev-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 14:50:50 2010 Return-Path: Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: (qmail 33087 invoked from network); 6 Apr 2010 14:50:50 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 6 Apr 2010 14:50:50 -0000 Received: (qmail 70477 invoked by uid 500); 6 Apr 2010 14:50:50 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 70284 invoked by uid 500); 6 Apr 2010 14:50:50 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 70276 invoked by uid 99); 6 Apr 2010 14:50:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 14:50:49 +0000 X-ASF-Spam-Status: No, hits=-1.3 required=10.0 tests=AWL,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [129.93.181.2] (HELO mathstat.unl.edu) (129.93.181.2) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 14:50:42 +0000 Received: from pcp089297pcs.unl.edu (pcp089297pcs.unl.edu [129.93.159.156]) (authenticated bits=0) by mathstat.unl.edu (8.13.8/8.13.8) with ESMTP id o36EoIDi024159 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for ; Tue, 6 Apr 2010 09:50:20 -0500 From: Brian Bockelman Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: multipart/signed; boundary=Apple-Mail-47-200442573; protocol="application/pkcs7-signature"; micalg=sha1 Subject: Re: HDFS Blockreport question Date: Tue, 6 Apr 2010 09:50:18 -0500 In-Reply-To: To: hdfs-dev@hadoop.apache.org References: Message-Id: X-Mailer: Apple Mail (2.1078) --Apple-Mail-47-200442573 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hey Jay, I think, if you're experienced in implementing transfer protocols, it is = not difficult to implement the HDFS wire protocol. As you point out, = they are subject to change between releases (especially between 0.20, = 0.21, and 0.22) and basically documented in fragments in the java source = code. At least, I looked at doing this for the read portions, and it = wasn't horrible. However, the *really hard part* is the client retry/recovery logic. = That's where a lot of the intelligence is, in very large classes, and = not incredibly well-documented. I've had lots of luck with scaling libhdfs - we average >20TB / day and = billions of I/O operations a day with it. I'd strongly advise not = re-inventing the wheel, unless it's for a research project. Brian On Apr 6, 2010, at 8:53 AM, Jay Booth wrote: > A pure C library to communicate with HDFS? >=20 > Certainly possible, but it would be a lot of work, and the HDFS wire > protocols are ad hoc, only somewhat documented and subject to change = between > releases right now so you'd be chasing a moving target. I'd try to = think of > another way to accomplish what you want to do before attempting a = client > reimplementation in C right now.. if you only need to talk to the = namenode > and not the datanodes it might be a little easier but still, lots of = work > that will probably be obsolete after another release or two. >=20 >=20 > On Tue, Apr 6, 2010 at 9:47 AM, Alberich de megres = wrote: >=20 >> Thanks! >>=20 >> I'm already using eclipse to browse the code. >> In this scenario, i could understand that java serializes the object >> through the network and its parameters. is that ok? >>=20 >> For example, if i want to make a pure C library (with no JNI >> interfaces).. is it possible/feasible? or it will be like to freeze >> the hell? >>=20 >> Thanks once again!!! >>=20 >>=20 >> On Sat, Apr 3, 2010 at 1:54 AM, Ryan Rawson = wrote: >>> If you look at the getProxy code it passes an "Invoker" (or = something >>> like that) which the proxy code uses to delegate calls TO. The >>> Invoker will call another class "Client" which has sub-classes like >>> Call, and Connection which wrap the actual java IO. This all lives = in >>> the org.apache.hadoop.ipc package. >>>=20 >>> Be sure to use a good IDE like IJ or Eclipse to browse the code, it >>> makes following all this stuff much easier. >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>> On Fri, Apr 2, 2010 at 4:39 PM, Alberich de megres >>> wrote: >>>> Hi again! >>>>=20 >>>> Anyone could help me? >>>> I could not understand how RPC class works. For me, only tries to >>>> instantiates a single interfaces with no declaration for some = methods >>>> like blockreport. But then it uses rpc.getproxy to get new class = wich >>>> send messages with name node. >>>>=20 >>>> I'm sorry for this silly question, but i am really lost at this = point. >>>>=20 >>>> Thanks for the patience. >>>>=20 >>>>=20 >>>>=20 >>>> On Fri, Apr 2, 2010 at 2:11 AM, Alberich de megres >>>> wrote: >>>>> Hi Jay! >>>>>=20 >>>>> thanks for the answear but i'm asking for what it works it sends? >>>>> blockreport is an interface in DatanodeProtocol that has no >>>>> declaration. >>>>>=20 >>>>> thanks! >>>>>=20 >>>>>=20 >>>>>=20 >>>>> On Thu, Apr 1, 2010 at 5:50 PM, Jay Booth = wrote: >>>>>> In DataNode: >>>>>> public DatanodeProtocol namenode >>>>>>=20 >>>>>> It's not a reference to an actual namenode, it's a wrapper for a >> network >>>>>> protocol created by that RPC.waitForProxy call -- so when it = calls >>>>>> namenode.blockReport, it's sending that information over RPC to = the >> namenode >>>>>> instance over the network >>>>>>=20 >>>>>> On Thu, Apr 1, 2010 at 5:50 AM, Alberich de megres < >> alberich2k5@gmail.com>wrote: >>>>>>=20 >>>>>>> Hi everyone! >>>>>>>=20 >>>>>>> sailing throught the hdfs source code that comes with hadoop = 0.20.2, >> i >>>>>>> could not understand how hdfs sends blockreport to nameNode. >>>>>>>=20 >>>>>>> As i can see, in >>>>>>> src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java we >>>>>>> create this.namenode interface with RPC.waitForProxy call (wich = i >>>>>>> could not understand which class it instantiates, and how it = works). >>>>>>>=20 >>>>>>> After that, datanode generates block list report = (blockListAsLongs) >>>>>>> with data.getBlockReport, and call = this.namenode.blockReport(..), >>>>>>> inside namenode.blockReport it calls again = namesystem.processReport. >>>>>>> This leads to an update of block lists inside nameserver. >>>>>>>=20 >>>>>>> But how it sends over the network this blockreport? >>>>>>>=20 >>>>>>> Anyone can point me some light? >>>>>>>=20 >>>>>>> thanks for all! >>>>>>> (and sorry for the newbie question) >>>>>>>=20 >>>>>>> Alberich >>>>>>>=20 >>>>>>=20 >>>>>=20 >>>>=20 >>>=20 >>=20 --Apple-Mail-47-200442573 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIICjCCA/gw ggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCgmS JomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y aXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAwMDBaFw0xMzAxMjUw ODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEg MB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDEw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYjYaPbCD5mtbiQb7Ka3y1qAm0ZcqKC FciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3KlwjtumTMtOtg35KZCknUd8KM4VGTSFdL VG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0 yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4Vd Gy0eIIPw1pfvYwxO36rm0S109qvbsNlaroPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3 rz9LAgMBAAGjgZ4wgZswDgYDVR0PAQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4E FgQUyhkdEo5upDhdQtQxDgjb2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeow DwYDVR0TAQH/BAUwAwEB/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzAN BgkqhkiG9w0BAQUFAAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLm ph5DBghZUVF8Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4v tQO1KDxgZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3Qghgk FIhmE1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0 gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBAowggLyoAMC AQICAwCB+zANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD Ew1ET0VHcmlkcyBDQSAxMB4XDTA5MDYwMjE5NDExM1oXDTEwMDYwMjE5NDExM1owYTETMBEGCgmS JomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8wDQYDVQQLEwZQZW9wbGUx HzAdBgNVBAMTFkJyaWFuIEJvY2tlbG1hbiA1MDQzMDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDPWEl7hBiuFRVBSY4SwvG0HpkCZi74a0BeD0tNARgxoQVJ7jhJjR3G4y8ino0/5axt 2EEfIWUE+DVpV37IWOQl8q/wdvicnhbfjByxBbq4sfWPLepU7+Kd8k1FKHRHermARn9VxEkFLrLB Gp7O5EX4mFHDaQy+Vv0thtA+m4qKoM+DA/8cOkJA5Rn6ZS/v/vtBzJh9HimVnhBx4+rw2cvKN+7r lKsm7qTn9TCZmrQ97CvBEXSkHS11m8vYF6ZwcTgSCJM0M9nnX5JilupQO1vDICXSUZeWX2xpsqeL x1PFGWgDaYXxFGtTRt2Qc9EPwf9Dr72xGPbKN8u5HylpOMDnAgMBAAGjgcIwgb8wEQYJYIZIAYb4 QgEBBAQDAgWgMA4GA1UdDwEB/wQEAwIF4DAfBgNVHSMEGDAWgBTKGR0Sjm6kOF1C1DEOCNvZjRcN XTAYBgNVHSAEETAPMA0GCyqGSIb3TAMHAQMAMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwu ZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4LmNybDAfBgNVHREEGDAWgRRiYm9ja2VsbUBj c2UudW5sLmVkdTANBgkqhkiG9w0BAQUFAAOCAQEAp6KjcWnfnH/MGlUkUWstE9gtPeymHp+2r4zI w8JXigncJh/8qpSZqBcVhD24WFowI95otblrKYNZKW9f2G/hWwDSxZFqHhCDxFO12vDthrzOc3EH CwypJPvIlZPt/E/x93XruzPxJwPz84DKKuPoJAMeNlADbd+92YtRr2y+VuMpgZaebMAoeCdWH8Cq Y8xheNMajf8uiImBbatDuCu7qRvhwgxsMNLHEt4h853K1Zc181RlFGXG1+uL/Q/8VeKiASiCu+7L 1zpfLg7OCr6rJHb5S7wU+CeAvzSqmyy0fd2mwPeiX7huK+Cw4UjaB3yGKItzWT+KQJnV//wcSrzZ dTGCAv0wggL5AgEBMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERP RUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3Jp ZHMgQ0EgMQIDAIH7MAkGBSsOAwIaBQCgggFiMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJ KoZIhvcNAQkFMQ8XDTEwMDQwNjE0NTAxOVowIwYJKoZIhvcNAQkEMRYEFPdmzwotyI+OhcXIgMvD C8KbKO3oMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT 8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE AxMNRE9FR3JpZHMgQ0EgMQIDAIH7MIGBBgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixk ARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBB dXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAIH7MA0GCSqGSIb3DQEBAQUABIIB ALCpmS/nKZIE3Q1fZJHJQf9C2oITpP+C3G1w4kIT/1wIfrgSYC/wCItmvX61+iE9IhwzFDL4loV+ OIbjAUMr3daQMscuCwRSQPyWsz2h8kgY5Axx+p6exd1vjV98Dsfk8Ot0cxxwYA/BZ55o+k0z1Yva TTBz1GbtneqpjDJnzY6HQF5Ghl+DCYybZ06CIk/Qs8lv5t8YXTpDippQ8xZKUzCTMAVtNCwva2sI FMNBL1JSxnxmuX/vh1xMnMGzVVwH0jJkeDhlMstitpHa3mlD+Z3zI67Z5uNmn9hNTYHWkgZR0dOw GMI3067GJ3FN8VhC0sfcuDFp+pUu2OwSpCOX6jEAAAAAAAA= --Apple-Mail-47-200442573--