hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-10850) KerberosAuthenticator should not do the SPNEGO handshake
Date Mon, 28 Jul 2014 16:03:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066711#comment-14066711
] 

Alejandro Abdelnur edited comment on HADOOP-10850 at 7/28/14 4:03 PM:
----------------------------------------------------------------------

I've spent a good few hours looking into this.

Making {{KerberosAuthenticator}} to use JDK SPNEGO is straight forward, simply removing code.
Still we want to continue using the {{AuthenticatedURL}} because it sets the hadoop-auth cookie
which avoids triggering SPENGO every time (and failing with doing a PUT/POST with a payload).

But there is problem, (IMO) a nasty bug in the JDK SPNEGO implementation that makes it unusable
in a reliable way.

The (Sun) JDK {{sun.net.www.protocol.http.HttpURLConnection}} class uses the {{NegotiateAuthentication}}
class to keep track of hostnames that support SPNEGO. 

The {{NegotiateAuthentication}} class uses a static {{HashMap<String, Boolean> supported}}
where it keeps if a hostname supports SPNEGO or not.

The first time the JDK gets a {{WWW-Authenticate: NEGOTIATE}} for hostname it will try to
obtain the Kerberos service ticket for {{HTTP/<hostname>}}.

If the service ticket is obtained, then the entry in the {{supported}} map will be {{(<hostname>,TRUE)}},
if the service ticket is not obtained the entry will be {{(<hostname>, FALSE)}}.

The {{supported}} map is never purged, this means a given hostname will be blacklisted for
the lifetime of the JVM (the singleton is kept in the bootstrap classloader, so it affects
the whole JVM).

The problem is that the service ticket may not be obtained for different reasons:

* JVM client is not within  Kerberos login context
* KDC is not available at the time of the call
* {{HTTP/<hostname>}} principal does not exist at the time of the call

All these reasons can be 'transient'. If they ever happen before a successful attempt, the
hostname is blacklisted for the lifetime of the JVM.

When running testcases, depending on the order testcases run, they pass or the fail.

It is also reasonable to say these 'transient' errors can happen in production in a long running
service (NN, Oozie).

The good thing is, and that is why {{AuthenticatedURL}}/{{KerberosAuthenticator}} works fine,
is that the {{HttpURLConnection}} checks if the user code is doing SPNEGO handling and lets
the user code handle it. And because {{KerberosAuthenticator}} does not blacklists hostnames,
things work and recover if necessary.

Based on these finding, I would close this JIRA and HADOO-10453 as invalid.


was (Author: tucu00):
I've spent a good few hours looking into this.

Making {{KerberosAuthenticator}} to use JDK SPNEGO is straight forward, simply removing code.
Still we want to continue using the {{AuthenticatedURL}} because it sets the hadoop-auth cookie
which avoids triggering SPENGO every time (and failing with doing a PUT/POST with a payload).

But there is problem, (IMO) a nasty bug in the JDK SPNEGO implementation that makes it unusable
in a reliable way.

The (Sun) JDK {{sun.net.www.protocol.http.HttpURLConnection}} class uses the {{NegotiateAuthentication}}
class to keep track of hostnames that support SPNEGO. 

The {{NegotiateAuthentication}} class uses a static {{HashMap<String, Boolean> supported}}
where it keeps if a hostname supports SPNEGO or not.

The first time the JDK gets a {{WWW-Authenticate: NEGOTIATE}} for hostname it will try to
obtain the Kerberos service ticket for {{HTTP/<hostname>}}.

If the service ticket is obtained, then the entry in the {{supported}} map will be {{(<hostname>,TRUE)}},
if the service ticket is not obtained the entry will be {{(<hostname>, FALSE)}}.

The {{supported}} map is never purged, this means a given hostname will be blacklisted for
the lifetime of the JVM (the singleton is kept in the bootstrap classloader, so it affects
the whole JVM).

The problem is that the service ticket may not be obtained for different reasons:

* JVM client is not within  Kerberos login context
* KDC is not available at the time of the call
* {{HTTP/<hostname>}} principal does not exist at the time of the call

All these reasons can be 'transient'. If they ever happen before a successful attempt, the
hostname is blacklisted for the lifetime of the JVM.

When running testcases, depending on the order testcases run, they pass or the fail.

It is also reasonable to say these 'transient' errors can happen in production in a long running
service (NN, Oozie).

The good thing is, and that is why {{AuthenticatedURL}}/{{KerberosAuthenticator}} works fine,
is that the {{HttpURLConnection}} checks if the user code is doing SPNEGO handling and lets
the user code handle it. And because {{KerberosAuthenticator}} does not blacklists hostnames,
things work and recover if necessary.

Based on these finding, I would close this JIRA and HADOO-10453 as invalids.

> KerberosAuthenticator should not do the SPNEGO handshake
> --------------------------------------------------------
>
>                 Key: HADOOP-10850
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10850
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 2.4.1
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>
> As mentioned in HADOOP-10453, the JDK automatically does a SPNEGO handshake when opening
a connection with a URL within a Kerberos login context, there is no need to do the SPNEGO
handshake in the {{KerberosAuthenticator}}, simply extract the auth token (hadoop-auth cookie)
and do the fallback if necessary.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message