Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 3 Feb 2017 02:47:52 +0000 (UTC)
From: "Duo Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12992881.1469615941000.22688.1486090072384@Atlassian.JIRA>
In-Reply-To: <JIRA.12992881.1469615941000@Atlassian.JIRA>
References: <JIRA.12992881.1469615941000@Atlassian.JIRA> <JIRA.12992881.1469615941416@jira-lw-us.apache.org>
Subject: [jira] [Reopened] (HADOOP-13433) Race in UGI.reloginFromKeytab
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 03 Feb 2017 02:48:01 -0000


     [ https://issues.apache.org/jira/browse/HADOOP-13433?page=3Dcom.atlass=
ian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Duo Zhang reopened HADOOP-13433:
--------------------------------

Reopen to attach patches for branch-2.8 and branch-2.7.

> Race in UGI.reloginFromKeytab
> -----------------------------
>
>                 Key: HADOOP-13433
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13433
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>            Reporter: Duo Zhang
>            Assignee: Duo Zhang
>             Fix For: 2.9.0, 3.0.0-alpha3
>
>         Attachments: HADOOP-13433-branch-2.patch, HADOOP-13433.patch, HAD=
OOP-13433-v1.patch, HADOOP-13433-v2.patch, HADOOP-13433-v4.patch, HADOOP-13=
433-v5.patch, HADOOP-13433-v6.patch, HBASE-13433-testcase-v3.patch
>
>
> This is a problem that has troubled us for several years. For our HBase c=
luster, sometimes the RS will be stuck due to
> {noformat}
> 2016-06-20,03:44:12,936 INFO org.apache.hadoop.ipc.SecureClient: Exceptio=
n encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExce=
ption: No valid credentials provided (Mechanism level: The ticket isn't for=
 us (35) - BAD TGS SERVER NAME)]
>         at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(=
GssKrb5Client.java:194)
>         at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnec=
t(HBaseSaslRpcClient.java:140)
>         at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setu=
pSaslConnection(SecureClient.java:187)
>         at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.acce=
ss$700(SecureClient.java:95)
>         at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.ru=
n(SecureClient.java:325)
>         at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.ru=
n(SecureClient.java:322)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroup=
Information.java:1781)
>         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth=
odAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
>         at org.apache.hadoop.hbase.security.User.call(User.java:607)
>         at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
>         at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(U=
ser.java:461)
>         at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setu=
pIOstreams(SecureClient.java:321)
>         at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseCli=
ent.java:1164)
>         at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:=
1004)
>         at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(Sec=
ureRpcEngine.java:107)
>         at $Proxy24.replicateLogEntries(Unknown Source)
>         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSo=
urce.shipEdits(ReplicationSource.java:962)
>         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSo=
urce.runLoop(ReplicationSource.java:466)
>         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSo=
urce.run(ReplicationSource.java:515)
> Caused by: GSSException: No valid credentials provided (Mechanism level: =
The ticket isn't for us (35) - BAD TGS SERVER NAME)
>         at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.=
java:663)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl=
.java:248)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl=
.java:180)
>         at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(=
GssKrb5Client.java:175)
>         ... 23 more
> Caused by: KrbException: The ticket isn't for us (35) - BAD TGS SERVER NA=
ME
>         at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:64)
>         at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
>         at sun.security.krb5.internal.CredentialsUtil.serviceCreds(Creden=
tialsUtil.java:294)
>         at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds=
(CredentialsUtil.java:106)
>         at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.=
java:557)
>         at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.=
java:594)
>         ... 26 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
>         at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
>         at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
>         at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:53)
>         at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:46)
>         ... 31 more=E2=80=8B
> {noformat}
> It rarely happens, but if it happens, the regionserver will be stuck and =
can never recover.
> Recently we added a log after a successful re-login which prints the priv=
ate credentials, and finally catched the direct reason. After a successful =
re-login, we have two kerberos tickets in the credentials, one is the TGT, =
and the other is a service ticket. The strange thing is that, the service t=
icket is placed before TGT. This breaks the assumption of jdk's kerberos li=
brary. See http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/s=
rc/share/classes/sun/security/jgss/krb5/Krb5InitCredential.java, the {{getT=
gt}} Method
> {code:title=3DKrb5InitCredential}
>             return AccessController.doPrivileged(
>                 new PrivilegedExceptionAction<KerberosTicket>() {
>                 public KerberosTicket run() throws Exception {
>                     // It's OK to use null as serverPrincipal. TGT is alm=
ost
>                     // the first ticket for a principal and we use list.
>                     return Krb5Util.getTicket(
>                         realCaller,
>                         clientPrincipal, null, acc);
>                         }});
> {code}
> So here, the library will use the service ticket as TGT to acquire a serv=
ice ticket, and KDC will reject the request since the 'TGT' does not start =
with 'krbtgt'. And it can never recover because in UGI, the re-login will c=
heck if there is a valid TGT first and no doubt, we have one...
> This usually happens when a secure connection initialization comes along =
with the re-login, and the end time indicates that the service ticket is ac=
quired by the previous TGT. Since UGI does not prevent doAs and re-login ha=
ppen at the same time, we believe that there is a race condition.
> After reading the code, we found a possible race condition.
> See http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/sh=
are/classes/sun/security/jgss/krb5/Krb5Context.java, the {{initSecContext}}=
 method, we will get TGT first, then check if there is already a service ti=
cket, if not, acquire a service ticket using the TGT, and put it into the c=
redentials.
> And in Krb5LoginModule.logout(the sun version), we will remove the kerber=
os tickets from the credentials first, and then destroy them.
> Here comes the race condition. Let T1 be the secure connection set up thr=
ead, T2 be the re-login thread.
> T1: get TGT
> T2: remove all tickets from credentials
> T1: check service ticket, none(since all tickets have been removed)
> T1: acquire a new service ticket using TGT and put it into the credential=
s
> T2: destroy all tickets
> T2: login, i.e., put a new TGT into the credentials.
> It is hard to write a UT to produce the problem because the racing code i=
s in jdk, which is not written by us...
> Suggestions are welcomed. Thanks.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org