hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gagan Brahmi <gaganbra...@gmail.com>
Subject HAWQ with Ranger KMS
Date Thu, 23 Mar 2017 18:15:25 GMT
Hi All,

Is there any known issue around running HAWQ in a cluster with Ranger KMS
over YARN?

Seems that HAWQ is not able to obtain containers when requesting it to YARN
ResourceManager.

The following is what I am seeing in the YARN RM logs:

---------------------

2017-03-23 10:56:30,816 INFO  hdfs.DFSClient
(DFSClient.java:getDelegationToken(1043)) - Created HDFS_DELEGATION_TOKEN
token 20049 for postgres on 192.168.59.104:8020
2017-03-23 10:56:30,889 WARN  security.DelegationTokenRenewer
(DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(895)) - Unable
to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
        at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1032)
        at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
        at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2298)
        at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:685)
        at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.obtainSystemTokensForUser(DelegationTokenRenewer.java:679)
        at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.requestNewHdfsDelegationToken(DelegationTokenRenewer.java:643)
        at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:488)
        at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77)
        at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:891)
        at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:868)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.UndeclaredThrowableException
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
        at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1014)
        ... 16 more
Caused by:
org.apache.hadoop.security.authentication.client.AuthenticationException:
Authentication failed, URL:
http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres&user.name=yarn,
status: 403, message: Forbidden
        at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:278)
        at
org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
        at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)
        at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
        at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
        at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
        at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
        at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1019)
        at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1014)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        ... 17 more

---------------------

The following is what I see in Ranger KMS log (kms.lo)

---------------------

2017-03-23 11:02:00,734 DEBUG LimitLatch - Counting
up[http-bio-9292-Acceptor-0] latch=7
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [uriBC] has
value [/kms/v1/]
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [semicolon] has
value [-1]
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [enc] has value
[ISO-8859-1]
2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Security checking request
OPTIONS /kms/v1/
2017-03-23 11:02:00,738 DEBUG RealmBase -   No applicable constraints
defined
2017-03-23 11:02:00,738 DEBUG AuthenticatorBase -  Not subject to any
constraint
2017-03-23 11:02:00,738 TRACE StandardWrapper -   Returning non-STM instance
2017-03-23 11:02:00,739 DEBUG Http11Protocol - Socket:
[org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State
out: [OPEN]
2017-03-23 11:02:00,758 DEBUG Http11Processor - Error parsing HTTP request
header
java.io.EOFException: Unexpected EOF read on the socket
        at
org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:169)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:990)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
        at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
2017-03-23 11:02:00,758 DEBUG Http11Protocol - Socket:
[org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State
out: [CLOSE
D]
2017-03-23 11:02:00,758 TRACE JIoEndpoint - Closing
socket:org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/
192.168.59.104,port=58547,localport=9292]

---------------------

The following is from the Ranger KMS access log:

---------------------

2017-03-23 11:02:00,738 UNAUTHENTICATED RemoteHost:192.168.59.104
Method:OPTIONS URL:
http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
tgres ErrorMsg:'Authentication required'
2017-03-23 11:02:00,786 UNAUTHENTICATED RemoteHost:192.168.59.104
Method:OPTIONS URL:
http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
tgres&user.name=yarn ErrorMsg:'Authentication required'

---------------------

The following is from the Ranger KMS audit log (kms-audit.log)

---------------------

192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS
/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
40gagan.com&doAs=postgres HTTP/1.1" 401 997
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS
/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
40gagan.com&doAs=postgres HTTP/1.1" 403 258
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS
/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
40gagan.com&doAs=postgres&user.name=yarn HTTP/1.1" 401 997
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS
/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%
40gagan.com&doAs=postgres&user.name=yarn HTTP/1.1" 403 258

---------------------


I have added the following proxyuser configuration in Ranger KMS as well:

hadoop.kms.proxyuser.postgres.users=*
hadoop.kms.proxyuser.postgres.hosts=*
hadoop.kms.proxyuser.yarn.users=*
hadoop.kms.proxyuser.yarn.hosts=*

The core-site.xml has the required proxyuser configuration as well:

hadoop.proxyuser.postgres.groups=*
hadoop.proxyuser.postgres.hosts=*
hadoop.proxyuser.yarn.groups=*
hadoop.proxyuser.yarn.hosts=*

But nothing seem to be working in this case here.

I would appreciate some inputs on this one.



Regards,
Gagan Brahmi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message