Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EEABA17464 for ; Wed, 27 May 2015 16:30:19 +0000 (UTC) Received: (qmail 21133 invoked by uid 500); 27 May 2015 16:30:19 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 21063 invoked by uid 500); 27 May 2015 16:30:19 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 20810 invoked by uid 99); 27 May 2015 16:30:19 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 May 2015 16:30:19 +0000 Date: Wed, 27 May 2015 16:30:19 +0000 (UTC) From: "Ambud Sharma (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-8485) Transparent Encryption Fails to work with Yarn/MapReduce MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561220#comment-14561220 ] Ambud Sharma commented on HDFS-8485: ------------------------------------ Seems like YARN is failing to get the delegation token to authenticate with Hadoop KMS. I am using a proxy user to launch the job on behalf of another user (hdfs/superadmin). > Transparent Encryption Fails to work with Yarn/MapReduce > -------------------------------------------------------- > > Key: HDFS-8485 > URL: https://issues.apache.org/jira/browse/HDFS-8485 > Project: Hadoop HDFS > Issue Type: Bug > Environment: RHEL-7, Kerberos 5 > Reporter: Ambud Sharma > Priority: Critical > Attachments: core-site.xml, hdfs-site.xml, kms-site.xml, mapred-site.xml, yarn-site.xml > > > Running a simple MapReduce job that writes to a path configured as an encryption zone throws exception > 11:26:26,343 INFO [org.apache.hadoop.mapreduce.Job] (pool-14-thread-1) Task Id : attempt_1432740034176_0001_m_000000_2, Status : FAILED > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) Error: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:424) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:710) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1358) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1457) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1442) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:400) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > 11:26:26,346 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at com.s3.ingestion.S3ImportMR$S3ImportMapper.map(S3ImportMR.java:112) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at com.s3.ingestion.S3ImportMR$S3ImportMapper.map(S3ImportMR.java:43) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at java.security.AccessController.doPrivileged(Native Method) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at javax.security.auth.Subject.doAs(Subject.java:422) > 11:26:26,347 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:418) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:413) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at java.security.AccessController.doPrivileged(Native Method) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at javax.security.auth.Subject.doAs(Subject.java:422) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > 11:26:26,348 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:413) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) ... 23 more > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at java.security.AccessController.doPrivileged(Native Method) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at javax.security.auth.Subject.doAs(Subject.java:422) > 11:26:26,349 ERROR [stderr] (pool-14-thread-1) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) > 11:26:26,350 ERROR [stderr] (pool-14-thread-1) ... 33 more > 11:26:26,350 ERROR [stderr] (pool-14-thread-1) -- This message was sent by Atlassian JIRA (v6.3.4#6332)