accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (ACCUMULO-3849) Proxy sets incorrect primary for SASL server transport
Date Fri, 22 May 2015 23:57:17 GMT

     [ https://issues.apache.org/jira/browse/ACCUMULO-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Elser resolved ACCUMULO-3849.
----------------------------------
    Resolution: Fixed

> Proxy sets incorrect primary for SASL server transport
> ------------------------------------------------------
>
>                 Key: ACCUMULO-3849
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3849
>             Project: Accumulo
>          Issue Type: Bug
>          Components: proxy
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>            Priority: Blocker
>             Fix For: 1.8.0, 1.7.1
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> A doozie for a Friday afternoon before a long weekend:
> On SuSE11, KerberosProxyIT was failing with the client unable to set up the SASL handshake.
> {noformat}
> 2015-05-20 06:27:44,670 [proxy.Proxy] INFO : Proxy server started on ip-172-31-5-57.ec2.internal:57147
> 2015-05-20 06:27:45,227 [transport.TSaslServerTransport] DEBUG: transport map does not
contain key
> 2015-05-20 06:27:45,232 [transport.TSaslServerTransport] DEBUG: Received start message
with status START
> 2015-05-20 06:27:45,232 [transport.TSaslServerTransport] DEBUG: Received mechanism name
'GSSAPI'
> 2015-05-20 06:27:45,248 [transport.TSaslTransport] ERROR: SASL negotiation failure
> javax.security.sasl.SaslException: Failure to initialize security context [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
credentails)]
> 	at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:125)
> 	at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
> 	at javax.security.sasl.Sasl.createSaslServer(Sasl.java:524)
> 	at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:140)
> 	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> 	at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> 	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> 	at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> 	at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:360)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
> 	at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos credentails)
> 	at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:87)
> 	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127)
> 	at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
> 	at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427)
> 	at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62)
> 	at sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
> 	at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:108)
> 	... 17 more
> 2015-05-20 06:27:45,254 [transport.TSaslServerTransport] DEBUG: failed to open server
transport
> org.apache.thrift.transport.TTransportException: Failure to initialize security context
> 	at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
> 	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
> 	at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> 	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> 	at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> 	at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:360)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
> 	at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> 	at java.lang.Thread.run(Thread.java:745)
> 2015-05-20 06:27:45,260 [server.TThreadPoolServer] ERROR: Error occurred during processing
of message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Failure
to initialize security context
> 	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> 	at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> 	at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:360)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
> 	at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException: Failure to initialize security
context
> 	at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
> 	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
> 	at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> 	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> 	... 11 more
> {noformat}
> So, the Thrift code is unable to actually use the KRB credentials we _know_ we logged
in with. Strange.
> Looking a bit earlier, we can see that we did log in.
> {noformat}
> 2015-05-20 06:27:44,498 [security.UserGroupInformation] INFO : Login successful for user
proxy/hostname@EXAMPLE.COM using keytab file /grid/0/hadoopqe/artifacts/accumulo/test/target/kerberos/keytabs/proxy.keytab
> 2015-05-20 06:27:44,498 [proxy.Proxy] INFO : Logged in as proxy/hostname@EXAMPLE.COM
> {noformat}
> So, for some reason, when we log in on SuSE, we somehow later dont' have the right credentials?
> Just after we log in, we start the Thrift server for the proxy
> {noformat}
> 2015-05-20 06:27:44,516 [rpc.TServerUtils] DEBUG: Instantiating SASL Thrift server
> 2015-05-20 06:27:44,524 [rpc.TServerUtils] INFO : Creating SASL thread pool thrift server
on listening on hostname:57147
> 2015-05-20 06:27:44,532 [rpc.TServerUtils] DEBUG: Logged in as proxy/hostname@EXAMPLE.COM
(auth:KERBEROS), creating TSaslServerTransport factory with accumulo/hostname
> {noformat}
> Hold up:
> {noformat}
> proxy/hostname@EXAMPLE.COM != accumulo/hostname
> {noformat}
> Turns out, when we created the ClientConfiguration for the ProxyServer, we didn't actually
set the kerberosPrimary (the client needs to know the 'primary' of the principal of the server
in which it's authenticating with). Somehow, on _every other OS and environment_ this didn't
error out like it should have. I have no explanation why.
> Sorry, SuSE. You did it right.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message