zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Irfan Hamid <iha...@salesforce.com>
Subject Re: ZK 3.4.6 creating malformed TGT
Date Wed, 24 Aug 2016 21:58:37 GMT
Sure. It was not very straightforward. I ended up setting the logging level
on the client side by adding the -Dlogging.level=DEBUG and
 -Dsun.security.krb5.debug=true. Additionally I set debug=true in the
jaas.conf.

With this I got log lines similar to the following on the client:

KrbException: Fail to create credential. (63) - No service creds
>>> Credentials acquireServiceCreds: no tgt; cannot get creds
>>> Credentials acquireServiceCreds: no tgt; searching backwards
msgType is 30
sname is krbtgt/B.C.NET <http://app.eng.sfdc.net/>
realm is A.B.C.NET <http://APP.ENG.SFDC.NET>
cname is zkcli
crealm is A.B.C.NET <http://APP.ENG.SFDC.NET>
error Message is Server not found in Kerberos database
error code is 7
suSec is 356619
sTime is Wed Aug 17 18:32:34 GMT 2016 1471458754000
cTime is Wed Feb 17 16:12:06 GMT 2010 1266423126000


The Internet had very little information about all that sname, realm, cname
and crealm etc. mean. However, a grep for krbtgt on the ZK sources led me
to Login.java and this code:

        Set<KerberosTicket> tickets =
subject.getPrivateCredentials(KerberosTicket.class);
        for(KerberosTicket ticket: tickets) {
            KerberosPrincipal server = ticket.getServer();
            if (server.getName().equals("krbtgt/" + server.getRealm() + "@"
+ server.getRealm())) {
                LOG.debug("Found tgt " + ticket + ".");
                return ticket;
            }
        }
        return null;

Logs didn't indicate that my code had gotten to this point but it did get
me thinking that this is a problem with my jaas or my krb5.conf since the
KerberTicket objects are being retrieved from a Java API. The man page on
krb5.conf gave me the next clues. Playing around with the [domain_realm]
and default_realm etc. got me to the solution.

Happy to help,
Irfan.

On Wed, Aug 24, 2016 at 12:37 PM, Patrick Hunt <phunt@apache.org> wrote:

> Hi Irfan, this is very helpful, appreciate you responding with your
> findings. It definitely will help folks in the future!
>
> Could you explain, how did you debug this issue and identify the correct
> changes? I'm interested to know how you were able to track this one down.
>
> Patrick
>
>
> On Thu, Aug 18, 2016 at 9:48 AM, Irfan Hamid <ihamid@salesforce.com>
> wrote:
>
> > Thanks Patrick,
> >
> > Actually it wasn't a DNS issue but I believe this is something that
> others
> > in the community could find useful since the issue had to do with default
> > realms domain-to-realm conversions and krb5.conf issues.
> >
> > Our setup has multiple domains (one for prod and one for internal testing
> > environments). I'm not a security expert so my terminology may be off.
> The
> > TGT is of the format krbtgt/REALM@REALM. In our krb5.conf we had:
> >
> > [domain_realms]
> >     .a.b.c = A.B.C
> >     a.b.c = A.B.C
> >     .b.c = B.C
> >     b.c = B.C
> >
> > All our machine names end in b.c, and only some end in a.b.c (the client
> > machines end in b.c). So the first realm that was being selected for the
> > krbtgt was B.C and the second was from [libdefaults] default_realm which
> > was A.B.C. I replaced the standard krb5.conf with a custom one that had:
> >
> > [domain_realms]
> >     .b.c = A.B.C
> >     b.c = A.B.C
> >
> > With this change and pointing to this file via jvm arg
> > java.security.krb5.conf everything worked fine.
> >
> > Thanks,
> > Irfan.
> >
> > On Wed, Aug 17, 2016 at 7:56 PM, Patrick Hunt <phunt@apache.org> wrote:
> >
> > > I'm not an expert but I believe this indicates an environmental issue,
> > see
> > > "service ticket not found in the subject" here:
> > > https://steveloughran.gitbooks.io/kerberos_and_
> hadoop/content/sections/
> > > errors.html
> > >
> > > Patrick
> > >
> > >
> > > On Tue, Aug 16, 2016 at 12:11 PM, Irfan Hamid <ihamid@salesforce.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm very certain this is due to an error in my jaas or krb5 conf
> files
> > > but
> > > > I can't seem to figure out where. My jaas.conf looks something like
> > this:
> > > >
> > > > Client {
> > > >   com.sun.security.auth.module.Krb5LoginModule required
> > > >   useKeyTab=true
> > > >   keyTab="/etc/zkcli_app_eng.keytab"
> > > >   storeKey=true
> > > >   useTicketCache=false
> > > >   debug=true
> > > >   principal="zkcli@AY.BEE.SEE.NET";
> > > > };
> > > >
> > > > But when ZK starts up it sends a TGT request as krbtgt/
> > > > BEE.SEE.NET@AY.BEE.SEE.NET, i.e., it shaves off the AY from the
> realm
> > > > name (
> > > > BEE.SEE.NET is a valid realm in our setup but not the one I want to
> > > use).
> > > > I
> > > > see the following log lines:
> > > >          msgType is 30
> > > >          sname is krbtgt/BEE.SEE.NET
> > > >          realm is AY.BEE.SEE.NET
> > > >          cname is zkcli
> > > >          crealm is AY.BEE.SEE.NET
> > > >          error Message is Server not found in Kerberos database
> > > >          error code is 7
> > > >          suSec is 157006
> > > >          sTime is Tue Aug 16 19:00:48 GMT 2016 1471374048000
> > > >          cTime is Fri Sep 30 18:19:26 GMT 2016 1475259566000
> > > >
> > > >
> > > > And a little earlier there was:
> > > > Credentials acquireServiceCreds: main loop: [0] tempService=krbtgt/
> > > > BEE.SEE.NET@AY.BEE.SEE.NET
> > > > Realm parseCapaths: no cfg entry
> > > > Realm doInitialParse: cRealm=[AY.BEE.SEE.NET], sRealm=[BEE.SEE.NET]
> > > > Service ticket not found in the subject
> > > > Found ticket for zkcli@AY.BEE.SEE.NET to go to krbtgt/
> > > > AY.BEE.SEE.NET@AY.BEE.SEE.NET expiring on Tue Aug 16 21:00:06 GMT
> 2016
> > > >
> > > > I looked in the source code and the place I see that might be
> relevant
> > is
> > > > Login.java where we have Login.getTGT() which tries to obtain a TGT
> by
> > > > seeing if there's a ticket in the Subject of the form
> > krbtgt/REALM@REALM.
> > > > However, that part doesn't even get called since I don't even see the
> > log
> > > > line at the of the Login thread "TGT refresh thread started".
> > > >
> > > > Any help would be much appreciated.
> > > >
> > > > Thanks,
> > > > Irfan.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message