hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-14265) AuthenticatedURL swallows the exception received from server.
Date Sun, 27 Aug 2017 20:24:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rushabh S Shah updated HADOOP-14265:
------------------------------------
    Description: 
While debugging some issue with kms server, we found out that AuthenticatedURL swallows the
original exception from server and constructed {{AuthenticationException}} with response code
and response message.

Below is the stack trace which didn't help in figuring out why the getDelegationTokens call
failed.
Due to lack of info logs on the kms server side also, this made the debugging even harder.
{noformat}
2017-03-23 16:32:10,364 ERROR [HiveServer2-Background-Pool: Thread-17795] [] exec.Task (TezTask.java:execute(197))
- Failed to execute tez graph.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1042)
        at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
        at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2444)
        at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:107)
        at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:86)
        at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:76)
        at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:301)
        at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:180)
        at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:911)
   ...
Caused by: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1954)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1023)
        ... 31 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication
failed, status: 403, message: Forbidden
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
        at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:377)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$5.run(KMSClientProvider.java:1028)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$5.run(KMSClientProvider.java:1023)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
        ... 32 more
{noformat}

Following is the relevant chunk of code from branch-2.8 but the code in trunk hasn't changed
much.
{code:title=AuthenticatedURL.java|borderStyle=solid}
  public static void extractToken(HttpURLConnection conn, Token token) throws IOException,
AuthenticationException {
    int respCode = conn.getResponseCode();
    if (notExpectedResponseCode) {
    } else {
      token.set(null);
      throw new AuthenticationException("Authentication failed, status: " +           conn.getResponseCode()
+ ", message: " + conn.getResponseMessage());
    }
  }
{code}

  was:
While debugging some issue with kms server, we found out that AuthenticatedURL swallows the
original exception from server and constructed {{AuthenticationException}} with response code
and response message.

Below is the stack trace which didn't help in figuring out why the getDelegationTokens call
failed.
Due to lack of info logs on the kms server side also, this made the debugging even harder.
{noformat}
2017-03-23 16:32:10,364 ERROR [HiveServer2-Background-Pool: Thread-17795] [] exec.Task (TezTask.java:execute(197))
- Failed to execute tez graph.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1042)
        at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
        at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2444)
        at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:107)
        at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:86)
        at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:76)
        at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:301)
        at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:180)
        at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:911)
   ...
Caused by: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1954)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1023)
        ... 31 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication
failed, status: 403, message: Forbidden
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
        at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:377)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$5.run(KMSClientProvider.java:1028)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$5.run(KMSClientProvider.java:1023)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
        ... 32 more
{noformat}

Following is the relevant chunk of code from branch-2.8 but the code in trunks hasn't changed
much.
{code:title=AuthenticatedURL.java|borderStyle=solid}
  public static void extractToken(HttpURLConnection conn, Token token) throws IOException,
AuthenticationException {
    int respCode = conn.getResponseCode();
    if (notExpectedResponseCode) {
    } else {
      token.set(null);
      throw new AuthenticationException("Authentication failed, status: " +           conn.getResponseCode()
+ ", message: " + conn.getResponseMessage());
    }
  }
{code}


> AuthenticatedURL swallows the exception received from server.
> -------------------------------------------------------------
>
>                 Key: HADOOP-14265
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14265
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Rushabh S Shah
>            Assignee: Rushabh S Shah
>
> While debugging some issue with kms server, we found out that AuthenticatedURL swallows
the original exception from server and constructed {{AuthenticationException}} with response
code and response message.
> Below is the stack trace which didn't help in figuring out why the getDelegationTokens
call failed.
> Due to lack of info logs on the kms server side also, this made the debugging even harder.
> {noformat}
> 2017-03-23 16:32:10,364 ERROR [HiveServer2-Background-Pool: Thread-17795] [] exec.Task
(TezTask.java:execute(197)) - Failed to execute tez graph.
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1042)
>         at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2444)
>         at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:107)
>         at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:86)
>         at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:76)
>         at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:301)
>         at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:180)
>         at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:911)
>    ...
> Caused by: java.lang.reflect.UndeclaredThrowableException
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1954)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1023)
>         ... 31 more
> Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException:
Authentication failed, status: 403, message: Forbidden
>         at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
>         at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
>         at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:377)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$5.run(KMSClientProvider.java:1028)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$5.run(KMSClientProvider.java:1023)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
>         ... 32 more
> {noformat}
> Following is the relevant chunk of code from branch-2.8 but the code in trunk hasn't
changed much.
> {code:title=AuthenticatedURL.java|borderStyle=solid}
>   public static void extractToken(HttpURLConnection conn, Token token) throws IOException,
AuthenticationException {
>     int respCode = conn.getResponseCode();
>     if (notExpectedResponseCode) {
>     } else {
>       token.set(null);
>       throw new AuthenticationException("Authentication failed, status: " +         
 conn.getResponseCode() + ", message: " + conn.getResponseMessage());
>     }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message