Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 31 Mar 2017 20:19:41 +0000 (UTC)
From: "Rushabh S Shah (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13060787.1490991540000.179578.1490991581534@Atlassian.JIRA>
In-Reply-To: <JIRA.13060787.1490991540000@Atlassian.JIRA>
References: <JIRA.13060787.1490991540000@Atlassian.JIRA> <JIRA.13060787.1490991540432@jira-lw-us.apache.org>
Subject: [jira] [Created] (HADOOP-14265) AuthenticatedURL swallows the
 exception received from server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 31 Mar 2017 20:19:46 -0000

Rushabh S Shah created HADOOP-14265:
---------------------------------------

             Summary: AuthenticatedURL swallows the exception received from server.
                 Key: HADOOP-14265
                 URL: https://issues.apache.org/jira/browse/HADOOP-14265
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Rushabh S Shah
            Assignee: Rushabh S Shah


While debugging some issue with kms server, we found out that AuthenticatedURL swallows the original exception from server and constructed {{AuthenticationException}} with response code and response message.

Below is the stack trace which didn't help in figuring out why the getDelegationTokens call failed.
Due to lack of info logs on the kms server side also, this made the debugging even harder.
{noformat}
2017-03-23 16:32:10,364 ERROR [HiveServer2-Background-Pool: Thread-17795] [] exec.Task (TezTask.java:execute(197)) - Failed to execute tez graph.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1042)
        at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
        at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2444)
        at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:107)
        at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:86)
        at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:76)
        at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:301)
        at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:180)
        at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:911)
   ...
Caused by: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1954)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1023)
        ... 31 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
        at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:377)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$5.run(KMSClientProvider.java:1028)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$5.run(KMSClientProvider.java:1023)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
        ... 32 more
{noformat}

Following is the relevant chunk of code from branch-2.8 but the code in trunks hasn't changed much.
{code:title=AuthenticatedURL.java|borderStyle=solid}
  public static void extractToken(HttpURLConnection conn, Token token) throws IOException, AuthenticationException {
    int respCode = conn.getResponseCode();
    if (notExpectedResponseCode) {
    } else {
      token.set(null);
      throw new AuthenticationException("Authentication failed, status: " +           conn.getResponseCode() + ", message: " + conn.getResponseMessage());
    }
  }
{code}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org