ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yusaku Sako (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-10977) HDFS Rebalance failed with IllegalArgumentException: Does not contain a valid host:port authority
Date Mon, 14 Sep 2015 22:14:47 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-10977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yusaku Sako updated AMBARI-10977:
---------------------------------
    Assignee: Mahadev konar

> HDFS Rebalance failed with IllegalArgumentException: Does not contain a valid host:port
authority
> -------------------------------------------------------------------------------------------------
>
>                 Key: AMBARI-10977
>                 URL: https://issues.apache.org/jira/browse/AMBARI-10977
>             Project: Ambari
>          Issue Type: Bug
>         Environment: ambari-2.1.0-376, hdp-2.3.0.0-1880, sles11sp3
>            Reporter: Zack Marsh
>            Assignee: Mahadev konar
>             Fix For: 2.1.0
>
>
> The HDFS Rebalance is failing with the following error messages:
> stderr:
> {code}
> 2015-05-06 12:31:46,656 - Error while executing command 'rebalancehdfs':
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 214, in execute
>     method(env)
>   File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py",
line 243, in rebalancehdfs
>     logoutput = False,
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148,
in __init__
>     self.env.run()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
152, in run
>     self.run_action(resource, action)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
118, in run_action
>     provider_action()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 269, in action_run
>     raise ex
> Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"'
; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'' returned 255.
May 6, 2015 12:31:46 PM  Balancing took 888.0 milliseconds
> 15/05/06 12:31:46 ERROR balancer.Balancer: Exiting balancer due an exception
> java.lang.IllegalArgumentException: Does not contain a valid host:port authority: jolokia1.labs.teradata.com
> 	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
> 	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
> 	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153)
> 	at org.apache.hadoop.hdfs.DFSUtil.getNameServiceUris(DFSUtil.java:1037)
> 	at org.apache.hadoop.hdfs.DFSUtil.getNsServiceRpcUris(DFSUtil.java:978)
> 	at org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:682)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:794)
> {code}
> stdout:
> {code}
> Starting balancer with threshold = 10
> Executing command ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"'
; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'
> 2015-05-06 12:31:43,096 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export
 PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"'
; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10''] {'logoutput':
False, 'on_new_line': handle_new_line}
> [balancer] May 6, 2015 12:31:46 PM [balancer]  [balancer] Balancing took 888.0 milliseconds[balancer]

> [balancer] 15/05/06 12:31:46 ERROR balancer.Balancer: Exiting balancer due an exception
> java.lang.IllegalArgumentException: Does not contain a valid host:port authority: jolokia1.labs.teradata.com
> 	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
> 	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
> 	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153)
> 	at org.apache.hadoop.hdfs.DFSUtil.getNameServiceUris(DFSUtil.java:1037)
> 	at org.apache.hadoop.hdfs.DFSUtil[balancer] .getNsServiceRpcUris(DFSUtil.java:978)
> 	at org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:682)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:794)
> 2015-05-06 12:31:46,656 - Error while executing command 'rebalancehdfs':
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 214, in execute
>     method(env)
>   File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py",
line 243, in rebalancehdfs
>     logoutput = False,
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148,
in __init__
>     self.env.run()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
152, in run
>     self.run_action(resource, action)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
118, in run_action
>     provider_action()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 269, in action_run
>     raise ex
> Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"'
; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'' returned 255.
May 6, 2015 12:31:46 PM  Balancing took 888.0 milliseconds
> 15/05/06 12:31:46 ERROR balancer.Balancer: Exiting balancer due an exception
> java.lang.IllegalArgumentException: Does not contain a valid host:port authority: jolokia1.labs.teradata.com
> 	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
> 	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
> 	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153)
> 	at org.apache.hadoop.hdfs.DFSUtil.getNameServiceUris(DFSUtil.java:1037)
> 	at org.apache.hadoop.hdfs.DFSUtil.getNsServiceRpcUris(DFSUtil.java:978)
> 	at org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:682)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:794)
> {code}
> The values of the namenode rpc address properties are as follows:
> {code}
> dfs.namenode.rpc-address = <NN1 FQDN>
> dfs.namenode.rpc-address.<CLUSTER-NAME>.nn1 = <NN1 FQDN>:8020
> dfs.namenode.rpc-address.<CLUSTER-NAME>.nn2 = <NN2FQDN>:8020
> {code}
> Setting the plain dfs.namenode.rpc-address property to the Active Namenode at port 8020
allows the rebalance to succeed.
> However, if this property is set to the Stand-by namenode at port 8020 the rebalance
fails with the error:
> {code}
> 2015-05-06 12:45:59,673 - Error while executing command 'rebalancehdfs':
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 214, in execute
>     method(env)
>   File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py",
line 243, in rebalancehdfs
>     logoutput = False,
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148,
in __init__
>     self.env.run()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
152, in run
>     self.run_action(resource, action)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
118, in run_action
>     provider_action()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 269, in action_run
>     raise ex
> Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"'
; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'' returned 252.
15/05/06 12:45:56 INFO balancer.Balancer: Using a threshold of 10.0
> 15/05/06 12:45:56 INFO balancer.Balancer: namenodes  = [hdfs://jolokia1.labs.teradata.com:8020,
hdfs://JOLOKIA]
> 15/05/06 12:45:56 INFO balancer.Balancer: parameters = Balancer.Parameters[BalancingPolicy.Node,
threshold=10.0, max idle iteration = 5, number of nodes to be excluded = 0, number of nodes
to be included = 0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being
Moved
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation
category READ is not supported in state standby
> 	at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1785)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1301)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getServerDefaults(FSNamesystem.java:1613)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getServerDefaults(NameNodeRpcServer.java:593)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getServerDefaults(ClientNamenodeProtocolServerSideTranslatorPB.java:383)
> 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> .  Exiting ...
> May 6, 2015 12:45:59 PM  Balancing took 3.127 seconds
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message