hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Loudongfeng <loudongf...@huawei.com>
Subject Distcp doesn't work when destination cluster not configured for ResourceManager (Hadoop 2.0)
Date Mon, 25 Feb 2013 13:08:58 GMT
For example,when I type this command:
bin/hadoop  distcp  hdfs://hacluster:8020/mr-history/done/2013/02/18/000000/  hdfs://hacluster-slave:8020/mr-history/done/2013/02/18/000000/

hdfs://hacluster-slave is the destination HDFS cluster in HA mode. when it's not configured
for resourcemanager, distcp will fail with the following error:
13/02/25 12:00:27 ERROR tools.DistCp: Exception encountered
java.io.IOException: Failed to run job : Unable to map logical nameservice URI 'hdfs://hacluster-slave'
to a NameNode. Local configuration does not have a failover proxy provider configured.
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:384)
        at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
        at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:148)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:113)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:368)


And RM's log has this:
2013-02-25 12:00:26,857 ERROR org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1235)
PriviledgedActionException as:mapred/hadoop@HADOOP.COM (auth:KERBEROS) cause:java.io.IOException:
Unable to map logical nameservice URI 'hdfs://hacluster-slave' to a NameNode. Local configuration
does not have a failover proxy provider configured.
2013-02-25 12:00:26,857 INFO  org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:290)
RMAppManager submit application exception
java.io.IOException: Unable to map logical nameservice URI 'hdfs://hacluster-slave' to a NameNode.
Local configuration does not have a failover proxy provider configured.
        at org.apache.hadoop.hdfs.DFSClient$Renewer.getNNProxy(DFSClient.java:875)
        at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:840)
        at org.apache.hadoop.security.token.Token.renew(Token.java:372)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:368)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
        ...  at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:280)
        ...

After I put remote cluster hacluster-slave's configraion into RM's hdfs-site.xml file and
restart RM, then distcp works.

My question is ,shall RM support reading configuration from client side when doing delegation
renew? Then no need to configure remote cluster configurations and restart RM.


Thank you!


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message