Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9B448200AE3 for ; Wed, 4 May 2016 10:26:26 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 99A8A1609FD; Wed, 4 May 2016 10:26:26 +0200 (CEST) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 211501609F1 for ; Wed, 4 May 2016 10:26:24 +0200 (CEST) Received: (qmail 59391 invoked by uid 500); 4 May 2016 08:26:22 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 59380 invoked by uid 99); 4 May 2016 08:26:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2016 08:26:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 09CFCC1EAA for ; Wed, 4 May 2016 08:26:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.178 X-Spam-Level: X-Spam-Status: No, score=-0.178 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_FONT_LOW_CONTRAST=0.001, HTML_MESSAGE=2, RP_MATCHES_RCVD=-2.079] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=hitec.lu; domainkeys=pass (1024-bit key) header.from=Adam.Cecile@hitec.lu header.d=hitec.lu Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id u8ejKhrc6tPX for ; Wed, 4 May 2016 08:26:18 +0000 (UTC) Received: from mail.hitec.lu (mail.hitec.lu [194.154.202.19]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id DCCB25F22C for ; Wed, 4 May 2016 08:26:17 +0000 (UTC) Received: from mail.hitec.lu (localhost.localdomain [127.0.0.1]) by mail.hitec.lu (Postfix) with ESMTP id A8F6DC0004A8; Wed, 4 May 2016 08:26:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=simple; d=hitec.lu; h=from:to:cc :subject:date:message-id:references:in-reply-to:content-type :mime-version; s=postfix; bh=wQdGpPhdZigcd8hUuHFG9uC1MCU=; b=UsT QttF2MyKg5uRU1WakOUOFprpMbjCFbt12IUQo6x8cWf1mqSSYJRQEO7Z4tSO+EK+ vswb8hgtu9E6LYo/598mHVBpNlrN+HWVe9hdiHocQfwXq+jZS4JwZsKH+BIz7VOA 3WtKXbDruKtOliC8FKjr6OUdSDZpMI8cbTF6RlJU= DomainKey-Signature: a=rsa-sha1; c=simple; d=hitec.lu; h=from:to:cc :subject:date:message-id:references:in-reply-to:content-type :mime-version; q=dns; s=postfix; b=H5PR+YWErAQqwt/OS57HF2rLAoLIV 4wLL4qE78WRRmPin34R7EF48hViP3crS6DNfGfx9u8VOya60gbG+uZ/8MIzLQRFT ZAXH5p7ifZ0jXVahea+4S3gleZdMIbK/h4wCQ2uOKCEeP9axq7JQCVF69OYnU4Zk P6lT63bQ0xb0OM= Received: from HI-SRV-05.hitec.lan (unknown [IPv6:2001:7e8:8192:150:7035:8ab:c94b:ff11]) by mail.hitec.lu (Postfix) with ESMTPS id 9DBB1C0004A4; Wed, 4 May 2016 08:26:15 +0000 (UTC) Received: from HI-SRV-05.hitec.lan (2001:7e8:8192:150::205) by HI-SRV-05.hitec.lan (2001:7e8:8192:150::205) with Microsoft SMTP Server (TLS) id 15.0.913.22; Wed, 4 May 2016 10:26:15 +0200 Received: from HI-SRV-05.hitec.lan ([fe80::7035:8ab:c94b:ff11]) by HI-SRV-05.hitec.lan ([fe80::7035:8ab:c94b:ff11%12]) with mapi id 15.00.0913.011; Wed, 4 May 2016 10:26:14 +0200 From: "Cecile, Adam" To: Sandeep Nemuri CC: "user@hadoop.apache.org" Subject: RE: NameNode HA from a client perspective Thread-Topic: NameNode HA from a client perspective Thread-Index: AQHRpc7luofUpAlBEk2usBt/7XBxJ5+oPGOAgAAxdYo= Date: Wed, 4 May 2016 08:26:13 +0000 Message-ID: References: , In-Reply-To: Accept-Language: fr-FR, lb-LU, en-US Content-Language: fr-FR X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.102.115] Content-Type: multipart/alternative; boundary="_000_c00e93a6197249699f2701c7cf6a180bHISRV05hiteclan_" MIME-Version: 1.0 archived-at: Wed, 04 May 2016 08:26:26 -0000 --_000_c00e93a6197249699f2701c7cf6a180bHISRV05hiteclan_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, I'm not sure to understand your answer, may I add a little piece of code: def _build_hdfs_url(self, hdfs_path, hdfs_operation, opt_query_param_tuples= =3D[]): """ :type hdfs_path: str :type hdfs_operation: str """ if not hdfs_path.startswith("/"): raise WebHdfsException("The web hdfs path must start with / but= found " + hdfs_path, None, None) url =3D 'http://' + self.host + ':' + str(self.port) + '/webhdfs/v1= ' + hdfs_path + '?user.name=3D' + self.user + '&op=3D' + hdfs_operation len_param =3D len(opt_query_param_tuples) for index in range(len_param): key_value =3D opt_query_param_tuples[index] url +=3D "&{}=3D{}".format(key_value[0], str(key_value[1])) return url Here is a plain python standard distribution function extracted from an app= : the problem here is "self.host", it has to be IP address ou DNS name of t= he NameNode, however I'd like to turn this into something dynamic resolving= to the current active master. Regards, Adam. ________________________________ De : Sandeep Nemuri Envoy? : mercredi 4 mai 2016 09:15 ? : Cecile, Adam Cc : user@hadoop.apache.org Objet : Re: NameNode HA from a client perspective I think you can simply use the nameservice (dfs.nameservices) which is defi= ned in hdfs-site.xml The hdfs client should be able to resolve the current active namenode and g= et the necessary information. Thanks, Sandeep Nemuri [https://mailfoogae.appspot.com/t?sender=3DabmhzYW5kZWVwNkBnbWFpbC5jb20%3D&= type=3Dzerocontent&guid=3D69e2c096-0009-4482-a881-df6dfb44434f]? On Wed, May 4, 2016 at 12:04 PM, Cecile, Adam > wrote: Hello All, I'd like to have a piece of advice regarding how my HDFS clients should han= dle the NameNode high availability feature. I have a complete setup running with ZKFC and I can see one active and one = standby NameNode. When I kill the active one, the standy gets active and wh= en the original one get back online it turns into a standby node, perfect. However I'm not sure how my client apps should handle this, a couple of ide= as: * Handle the bad HTTP code from standby node to switch to the other one * Integrate Zookeeper client to query for the current active node * Hack something like a shared-ip linked to the active node Then I'll have to handle a switch that may occurs during the execution of a= client app: should I just crash and rely on the cluster to restart the job= . Thanks in advance, Best regards from Luxembourg.? -- Regards Sandeep Nemuri --_000_c00e93a6197249699f2701c7cf6a180bHISRV05hiteclan_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,


I'm not sure to understand your answer, may I add a little piece of = ;code:


def _build_hdfs_url(self, hdfs_path, hdfs_operation, opt_query_param_tuples=3D[]):=0A=
        """=0A=
        :type hdfs_path: str=
=0A=
        :type hdfs_operation: str=0A=
        """=0A=
        if not hdfs_path.startswith("/"):=0A=
            raise WebHdfsException("The web hdfs path must start=
 with / but found " + hdfs_path, =
None, None)=0A=
=0A=
        url =3D 'http://' + self.host + ':' + str(self=
.port) + '/webhdfs/v1' + hdfs_path +=
 '?user.name=3D' + self.user + '&op=3D' + hdfs_operation<=
/span>=0A=
        len_param =3D len(opt_qu=
ery_param_tuples)=0A=
        for index in range(len_param):=0A=
            key_value =3D opt_query_param_tuples[inde=
x]=0A=
            url +=3D "=
;&{}=3D{}".format=
(key_value[0], str(key_value[1]))=0A=
        return url


Here is a plain python standard distribution function extracted from an = app: the problem here is "self.host", it has to be IP address ou = DNS name of the NameNode, however I'd like to turn this into something dyna= mic resolving to the current active master.


Regards, Adam.


De : Sandeep Nemuri <nhs= andeep6@gmail.com>
Envoyé : mercredi 4 mai 2016 09:15
À : Cecile, Adam
Cc : user@hadoop.apache.org
Objet : Re: NameNode HA from a client perspective
 
I think you ca= n simply use the nameservice (dfs.names= ervices) which is defined in hdfs-site.xml
The hdfs client should be able to resolve the current active name= node and get the necessary information.

Thanks,
Sandeep Nemuri
ᐧ

On Wed, May 4, 2016 at 12:04 PM, Cecile, Adam <Adam.Cecile@h= itec.lu> wrote:

Hello All,

I'd like to have a piece of advice regarding = how my HDFS clients should handle the NameNode high availability feature.
I have a complete setup running with ZKFC and= I can see one active and one standby NameNode. When I kill the active one, the standy gets active and when the = original one get back online it turns into a standby node, perfect.<= br style=3D"color:rgb(40,40,40); font-family:'Segoe UI WPC','Segoe UI',Taho= ma,'Microsoft Sans Serif',Verdana,sans-serif; font-size:13.3333px; backgrou= nd-color:rgb(255,255,255)">
However I'm not sure how my client apps shoul= d handle this, a couple of ideas:
* Handle the bad HTTP code from standby node = to switch to the other one
* Integrate Zookeeper client to query for the= current active node
* Hack something like a shared-ip linked to t= he active node

Then I'll have to handle a switch that may oc= curs during the execution of a client app: should I just crash and rely on the cluster to restart the job.


Thanks in advance,

Best regards from Luxembourg.​




--
  Regards
  Sandeep Nemuri
--_000_c00e93a6197249699f2701c7cf6a180bHISRV05hiteclan_--