hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Jie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13214) RBF: Configuration on Router conflicts with client side configuration
Date Fri, 02 Mar 2018 06:21:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383236#comment-16383236

Tao Jie commented on HDFS-13214:

[~elgoiri] [~ywskycn] [~linyiqun] thank you for your response.
In our internal setup, we configure dfs.nameservice.id.
 I have made brief test about HA/non-HA configuration, once we don't specify {{dfs.nameservice.id}},
the Exception occurs no matter whether HA is enabled. So I don't think HA/non-HA mode is directly
related to this issue.
In current code logic, we try to find the local namenode host from the configuration. So I
think we should set {{dfs.nameservice.id}} to {{ns1}} or {{ns2}} rather than {{ns-fed}}. Otherwise
the Router would think itself as the local namenode by mistake.
Today property {{dfs.nameservice.id}} is not a necessary one in a federation cluster (HA or
non-HA), right?
1, We can complete the document and ensure {{dfs.nameservice.id}} must be specified on Router
2, Improve the logic of finding the local namenode address in case of {{dfs.nameservice.id}}
not be specified.
Please correct me if I am wrong.

> RBF: Configuration on Router conflicts with client side configuration
> ---------------------------------------------------------------------
>                 Key: HDFS-13214
>                 URL: https://issues.apache.org/jira/browse/HDFS-13214
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: 2.9.0
>            Reporter: Tao Jie
>            Priority: Major
> In a typical router-based federation cluster, hdfs-site.xml is supposed to be:
> {code}
> <property>
>     <name>dfs.nameservices</name>
>     <value>ns1,ns2,ns-fed</value>
>   </property>
>   <property>
>     <name>dfs.ha.namenodes.ns-fed</name>
>     <value>r1,r2</value>
>   </property>
>   <property>
>     <name>dfs.namenode.rpc-address.ns1</name>
>     <value>host1:8020</value>
>   </property>
>   <property>
>     <name>dfs.namenode.rpc-address.ns2</name>
>     <value>host2:8020</value>
>   </property>
>   <property>
>     <name>dfs.namenode.rpc-address.ns-fed.r1</name>
>     <value>host1:8888</value>
>   </property>
>   <property>
>     <name>dfs.namenode.rpc-address.ns-fed.r2</name>
>     <value>host2:8888</value>
>   </property>
> {code}
> {{dfs.ha.namenodes.ns-fed}} here is used for client to access the Router. However with
this configuration on server node, Router fails to start with error:
> {code}
> org.apache.hadoop.HadoopIllegalArgumentException: Configuration has multiple addresses
that match local node's address. Please configure the system with dfs.nameservice.id and dfs.ha.namenode.id
>         at org.apache.hadoop.hdfs.DFSUtil.getSuffixIDs(DFSUtil.java:1198)
>         at org.apache.hadoop.hdfs.DFSUtil.getNameServiceId(DFSUtil.java:1131)
>         at org.apache.hadoop.hdfs.DFSUtil.getNamenodeNameServiceId(DFSUtil.java:1086)
>         at org.apache.hadoop.hdfs.server.federation.router.Router.createLocalNamenodeHearbeatService(Router.java:466)
>         at org.apache.hadoop.hdfs.server.federation.router.Router.createNamenodeHearbeatServices(Router.java:423)
>         at org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:199)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>         at org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> 2018-03-01 18:05:56,208 ERROR org.apache.hadoop.hdfs.server.federation.router.DFSRouter:
Failed to start router
> {code}
> Then the router tries to find the local namenode, multiple properties: {{dfs.namenode.rpc-address.ns1}},
{{dfs.namenode.rpc-address.ns-fed.r1}} match the local address.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message