hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1973) HA: HDFS clients must handle namenode failover and switch over to the new active namenode.
Date Wed, 21 Sep 2011 01:49:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109200#comment-13109200

Todd Lipcon commented on HDFS-1973:

- not sure if removing the DFSClient(Configuration conf) constructor is a good idea - it was
deprecated at 0.21 but since we haven't had a well-adopted release since then, seems we should
keep it around. (it's trivial to keep, right?)
- some log messages left in: LOG.info("address of nn1: " + conf.get("dfs.atm.nn1"))
- some of the lines seem to have a lot of columns - try to wrap at 80?


-  DFSClient(InetSocketAddress nameNodeAddr, ClientProtocol rpcNamenode,
+  DFSClient(URI nameNodeAddr, ClientProtocol rpcNamenode,
maybe rename this parameter to {{nameNodeUri}}?

+  public static final String  DFS_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX = "dfs.client.failover.proxy.provider";
Shouldn't that have a "." at the end? It seems authorities are appended without a "." in the

+    } catch (RuntimeException e) {
+      if (e.getCause() instanceof ClassNotFoundException) {
+        return null;
why? It seems we should at least be logging a WARN if this is configured to point to a class
that can't be loaded, if not re-throwing as IOE.


- I'm not sure that o.a.h.h.protocol is the right package for ConfiguredFailoverProxyProvider
- maybe it belongs in the HA package?
- missing license headers on many of the new files

+      InetSocketAddress first = NameNode.getAddress(
+          new URI(conf.get(CONFIGURED_NAMENODE_ADDRESS_FIRST)).getAuthority());
+      InetSocketAddress second = NameNode.getAddress(
+          new URI(conf.get(CONFIGURED_NAMENODE_ADDRESS_SECOND)).getAuthority());
these will throw NPE in the case of misconfiguration. Should at least throw an exception indicating
what the mistaken config is.

Also, it seems more sensible to have a single config like "dfs.ha.namenode.addresses" with
comma-separated URIs, since there's nothing explicitly tied to having exactly 2 NNs here,

A general note: this failover proxy provider doesn't get passed enough info to choose different
configs based on the "logical URI". For example, if I have two clusters, {{foo}} and {{bar}}
I'd expect to be able to configure {{dfs.ha.namenodes.foo}} separately from {{dfs.ha.namenodes.bar}}
after setting both of {{dfs.client.failover.proxy.provider.foo|bar}}.

- Can we separate the addition of Idempotent annotations to a different patch?

+  // This test should probably be made to fail if a client fails over to talk to
+  // an NN with a different block pool id.
file a followup JIRA or use "TODO" so we don't forget about this case?

> HA: HDFS clients must handle namenode failover and switch over to the new active namenode.
> ------------------------------------------------------------------------------------------
>                 Key: HDFS-1973
>                 URL: https://issues.apache.org/jira/browse/HDFS-1973
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Suresh Srinivas
>            Assignee: Aaron T. Myers
>         Attachments: HDFS-1973-HDFS-1623.patch, hdfs-1973.0.patch
> During failover, a client must detect the current active namenode failure and switch
over to the new active namenode. The switch over might make use of IP failover or some thing
more elaborate such as zookeeper to discover the new active.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message