falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Gresch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-505) Replication Job throws GSSException on secure cluster
Date Thu, 15 Jan 2015 15:46:34 GMT

    [ https://issues.apache.org/jira/browse/FALCON-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278829#comment-14278829
] 

Aaron Gresch commented on FALCON-505:
-------------------------------------

I ran into this as well on Falcon 0.6 trying to do replication on our secure clusters, and
just found this JIRA.  I had hacked the following change to oozie/src/main/resources/action/feed/replication-action.xml
and  oozie/src/main/resources/action/pre-process.xml, and that solved the problem:

<property>
                <name>oozie.launcher.mapreduce.job.hdfs-servers</name>
                <value>webhdfs://namenodeA.mycompany.com,webhdfs://namenodeB.mycompany.com</value>
</property>

This link provides documentation stating this property is necessary for distcp to work between
secure clusters:

http://oozie.apache.org/docs/4.0.0/DG_DistCpActionExtension.html


>  Replication Job throws GSSException on secure cluster 
> -------------------------------------------------------
>
>                 Key: FALCON-505
>                 URL: https://issues.apache.org/jira/browse/FALCON-505
>             Project: Falcon
>          Issue Type: Bug
>          Components: replication
>    Affects Versions: 0.5
>         Environment: Hadoop2/YARN (both source and target clusters)
> Security enabled
>            Reporter: Venkat R
>
> Replication job launched on target cluster by oozie a workflow throws GSSException when
it tries to access the source cluster HDFS using webhdfs (as well as hftp).
> Both the source and target cluster oozie instances have the oozie-site.xml pointing to
all the hadoop cluster configs they access (See first comment of JIRA: https://issues.apache.org/jira/browse/FALCON-389)
> It seems the Target cluster oozie coordinator instance was able to access the source
clusters HDFS, but from the job running in the clutser node.
> But, it works if I add the following property to the oozie/conf/hadoop-conf-cluster-1/mapred-site.xml:
>  <property>
>      <name>mapreduce.job.hdfs-servers</name>
>      <value>webhdfs://grid1nn01.grid.example.com,webhdfs://gird2nn01.grid.example.com</value>
>   </property>
> this enabled grid1 to do webhdfs calls to grid2 and vice-versa. In the absence, it throws
authentication errors. 
> It seems Oozie needs to get tokens for both the clusters before it can kick off the Falcon
job that does the distcp.
> It may be possible to add this property to the generated Oozie bundle by Falcon.
> Exception stacktrace:
> Failing Oozie Launcher, Main class [org.apache.falcon.latedata.LateDataHandler], main()
threw exception, Authentication failed, url=http://gridnn01.grid.example.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN&user.name=veramach
> java.io.IOException: Authentication failed, url=http://gridnn01.grid.example.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN&user.name=veramach
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:490)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:531)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:424)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:953)
> at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:143)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:227)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:381)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:402)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathRunner.getUrl(WebHdfsFileSystem.java:652)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:485)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:531)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:424)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:678)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:689)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:238)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1624)
> at org.apache.falcon.latedata.LateDataHandler.usage(LateDataHandler.java:269)
> at org.apache.falcon.latedata.LateDataHandler.getFileSystemUsageMetric(LateDataHandler.java:252)
> at org.apache.falcon.latedata.LateDataHandler.computeStorageMetric(LateDataHandler.java:224)
> at org.apache.falcon.latedata.LateDataHandler.computeMetrics(LateDataHandler.java:170)
> at org.apache.falcon.latedata.LateDataHandler.run(LateDataHandler.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.falcon.latedata.LateDataHandler.main(LateDataHandler.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
> at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:232)
> at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:164)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.openHttpUrlConnection(WebHdfsFileSystem.java:475)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$200(WebHdfsFileSystem.java:431)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:457)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:454)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getHttpUrlConnection(WebHdfsFileSystem.java:453)
> at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:487)
> ... 36 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
> ... 48 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message