Mailing-List: contact dev-help@hawq.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hawq.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CALqDVuYBOTFOt3TaM4PSyuQc8yqsm7349eP74wa8ROhJ6cEBZg@mail.gmail.com>
References: <CALqDVuYBOTFOt3TaM4PSyuQc8yqsm7349eP74wa8ROhJ6cEBZg@mail.gmail.com>
From: Gagan Brahmi <gaganbrahmi@gmail.com>
Date: Thu, 23 Mar 2017 12:46:18 -0700
Message-ID: <CALqDVubVa4OEFjWmTPLAEkSgx5Ek5NmA2NXwL=oo3D-m4VZDWw@mail.gmail.com>
Subject: Re: HAWQ with Ranger KMS
To: dev <dev@hawq.incubator.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c13140e7ba872054b6b219e
archived-at: Thu, 23 Mar 2017 19:51:41 -0000

--94eb2c13140e7ba872054b6b219e
Content-Type: text/plain; charset=UTF-8

This worked for me finally after Vineet's suggestion via HCC.

https://community.hortonworks.com/questions/90662/hawq-issues-with-ranger-kms.html

Thank you all.


Regards,
Gagan Brahmi

On Thu, Mar 23, 2017 at 11:15 AM, Gagan Brahmi <gaganbrahmi@gmail.com>
wrote:

> Hi All,
>
> Is there any known issue around running HAWQ in a cluster with Ranger KMS
> over YARN?
>
> Seems that HAWQ is not able to obtain containers when requesting it to
> YARN ResourceManager.
>
> The following is what I am seeing in the YARN RM logs:
>
> ---------------------
>
> 2017-03-23 10:56:30,816 INFO  hdfs.DFSClient (DFSClient.java:getDelegationToken(1043))
> - Created HDFS_DELEGATION_TOKEN token 20049 for postgres on
> 192.168.59.104:8020
> 2017-03-23 10:56:30,889 WARN  security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(895)) - Unable
> to add the application to the delegation token renewer.
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.
> addDelegationTokens(KMSClientProvider.java:1032)
>         at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExte
> nsion.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.
> addDelegationTokens(DistributedFileSystem.java:2298)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:685)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:680)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1724)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer.obtainSystemTokensForUser(
> DelegationTokenRenewer.java:679)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer.requestNewHdfsDelegationToken(
> DelegationTokenRenewer.java:643)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:
> 488)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer$DelegationTokenRenewerRunnable.
> handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:891)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.
> DelegationTokenRenewer$DelegationTokenRenewerRunnable
> .run(DelegationTokenRenewer.java:868)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.UndeclaredThrowableException
>         at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1742)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.
> addDelegationTokens(KMSClientProvider.java:1014)
>         ... 16 more
> Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException:
> Authentication failed, URL: http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
> 40gagan.com&doAs=postgres&user.name=yarn, status: 403, message: Forbidden
>         at org.apache.hadoop.security.authentication.client.
> AuthenticatedURL.extractToken(AuthenticatedURL.java:278)
>         at org.apache.hadoop.security.authentication.client.
> PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
>         at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.
> java:132)
>         at org.apache.hadoop.security.authentication.client.
> KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)
>         at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.
> java:132)
>         at org.apache.hadoop.security.authentication.client.
> AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
>         at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticator.doDelegationTokenOperation(
> DelegationTokenAuthenticator.java:298)
>         at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticator.getDelegationToken(
> DelegationTokenAuthenticator.java:170)
>         at org.apache.hadoop.security.token.delegation.web.
> DelegationTokenAuthenticatedURL.getDelegationToken(
> DelegationTokenAuthenticatedURL.java:371)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(
> KMSClientProvider.java:1019)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(
> KMSClientProvider.java:1014)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1724)
>         ... 17 more
>
> ---------------------
>
> The following is what I see in Ranger KMS log (kms.lo)
>
> ---------------------
>
> 2017-03-23 11:02:00,734 DEBUG LimitLatch - Counting
> up[http-bio-9292-Acceptor-0] latch=7
> 2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [uriBC] has
> value [/kms/v1/]
> 2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [semicolon] has
> value [-1]
> 2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [enc] has value
> [ISO-8859-1]
> 2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Security checking
> request OPTIONS /kms/v1/
> 2017-03-23 11:02:00,738 DEBUG RealmBase -   No applicable constraints
> defined
> 2017-03-23 11:02:00,738 DEBUG AuthenticatorBase -  Not subject to any
> constraint
> 2017-03-23 11:02:00,738 TRACE StandardWrapper -   Returning non-STM
> instance
> 2017-03-23 11:02:00,739 DEBUG Http11Protocol - Socket: [
> org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
> 192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State
> out: [OPEN]
> 2017-03-23 11:02:00,758 DEBUG Http11Processor - Error parsing HTTP request
> header
> java.io.EOFException: Unexpected EOF read on the socket
>         at org.apache.coyote.http11.Http11Processor.
> setRequestLineReadTimeout(Http11Processor.java:169)
>         at org.apache.coyote.http11.AbstractHttp11Processor.process(
> AbstractHttp11Processor.java:990)
>         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.
> process(AbstractProtocol.java:625)
>         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.
> run(JIoEndpoint.java:318)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
> TaskThread.java:61)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-03-23 11:02:00,758 DEBUG Http11Protocol - Socket: [
> org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
> 192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State
> out: [CLOSE
> D]
> 2017-03-23 11:02:00,758 TRACE JIoEndpoint - Closing
> socket:org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
> 192.168.59.104,port=58547,localport=9292]
>
> ---------------------
>
> The following is from the Ranger KMS access log:
>
> ---------------------
>
> 2017-03-23 11:02:00,738 UNAUTHENTICATED RemoteHost:192.168.59.104
> Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
> tgres ErrorMsg:'Authentication required'
> 2017-03-23 11:02:00,786 UNAUTHENTICATED RemoteHost:192.168.59.104
> Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
> tgres&user.name=yarn ErrorMsg:'Authentication required'
>
> ---------------------
>
> The following is from the Ranger KMS audit log (kms-audit.log)
>
> ---------------------
>
> 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres
> HTTP/1.1" 401 997
> 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres
> HTTP/1.1" 403 258
> 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com
> &doAs=postgres&user.name=yarn HTTP/1.1" 401 997
> 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=
> GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com
> &doAs=postgres&user.name=yarn HTTP/1.1" 403 258
>
> ---------------------
>
>
> I have added the following proxyuser configuration in Ranger KMS as well:
>
> hadoop.kms.proxyuser.postgres.users=*
> hadoop.kms.proxyuser.postgres.hosts=*
> hadoop.kms.proxyuser.yarn.users=*
> hadoop.kms.proxyuser.yarn.hosts=*
>
> The core-site.xml has the required proxyuser configuration as well:
>
> hadoop.proxyuser.postgres.groups=*
> hadoop.proxyuser.postgres.hosts=*
> hadoop.proxyuser.yarn.groups=*
> hadoop.proxyuser.yarn.hosts=*
>
> But nothing seem to be working in this case here.
>
> I would appreciate some inputs on this one.
>
>
>
> Regards,
> Gagan Brahmi
>

--94eb2c13140e7ba872054b6b219e--