falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkat R (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FALCON-505) Replication Job throws GSSException on secure cluster
Date Tue, 15 Jul 2014 18:40:05 GMT
Venkat R created FALCON-505:
-------------------------------

             Summary:  Replication Job throws GSSException on secure cluster 
                 Key: FALCON-505
                 URL: https://issues.apache.org/jira/browse/FALCON-505
             Project: Falcon
          Issue Type: Bug
          Components: replication
    Affects Versions: 0.5
         Environment: Hadoop2/YARN (both source and target clusters)
Security enabled
            Reporter: Venkat R


Replication job launched on target cluster by oozie a workflow throws GSSException when it
tries to access the source cluster HDFS using webhdfs (as well as hftp).

Both the source and target cluster oozie instances have the oozie-site.xml pointing to all
the hadoop cluster configs they access (See first comment of JIRA: https://issues.apache.org/jira/browse/FALCON-389)

It seems the Target cluster oozie coordinator instance was able to access the source clusters
HDFS, but from the job running in the clutser node.

But, it works if I add the following property to the oozie/conf/hadoop-conf-cluster-1/mapred-site.xml:

 <property>
     <name>mapreduce.job.hdfs-servers</name>
     <value>webhdfs://grid1nn01.grid.linkedin.com,webhdfs://gird2nn01.grid.linkedin.com</value>
  </property>

this enabled grid1 to do webhdfs calls to grid2 and vice-versa. In the absence, it throws
authentication errors. 

It seems Oozie needs to get tokens for both the clusters before it can kick off the Falcon
job that does the distcp.

It may be possible to add this property to the generated Oozie bundle by Falcon.



Exception stacktrace:

Failing Oozie Launcher, Main class [org.apache.falcon.latedata.LateDataHandler], main() threw
exception, Authentication failed, url=http://eat1-nertznn01.grid.linkedin.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN&user.name=veramach
java.io.IOException: Authentication failed, url=http://eat1-nertznn01.grid.linkedin.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN&user.name=veramach
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:490)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:531)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:424)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:953)
at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:143)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:227)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:381)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:402)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathRunner.getUrl(WebHdfsFileSystem.java:652)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:485)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:531)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:424)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:678)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:689)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:238)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1624)
at org.apache.falcon.latedata.LateDataHandler.usage(LateDataHandler.java:269)
at org.apache.falcon.latedata.LateDataHandler.getFileSystemUsageMetric(LateDataHandler.java:252)
at org.apache.falcon.latedata.LateDataHandler.computeStorageMetric(LateDataHandler.java:224)
at org.apache.falcon.latedata.LateDataHandler.computeMetrics(LateDataHandler.java:170)
at org.apache.falcon.latedata.LateDataHandler.run(LateDataHandler.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.falcon.latedata.LateDataHandler.main(LateDataHandler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:232)
at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:164)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.openHttpUrlConnection(WebHdfsFileSystem.java:475)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$200(WebHdfsFileSystem.java:431)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:457)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:454)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getHttpUrlConnection(WebHdfsFileSystem.java:453)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:487)
... 36 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any
Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
... 48 more




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message