hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maria <linanmengxia...@126.com>
Subject Re:Re: Re:Re:Re:Re: How to access linux kerberosed hive from windows eclipse workspace?
Date Tue, 05 Jul 2016 01:59:04 GMT

Yup,yesterday I started to realize that The renewal is a  principal level setting. I hava
fixed renew time in KDC kdc.conf. Do as Aviral said, I enable kerberos logs with     
    "-Dsun.security.krb5.debug=true" , more error info printed out:
------------------------------------------------------------------------------------------------
Java config name: E:\Program Files (x86)\Java\jre7\lib\security\krb5.conf
Loaded from Java config
Java config name: E:\Program Files (x86)\Java\jre7\lib\security\krb5.conf
Loaded from Java config
>>> KdcAccessibility: reset
>>> KdcAccessibility: reset
>>> KeyTabInputStream, readName(): HADOOP.COM
>>> KeyTabInputStream, readName(): hive
>>> KeyTabInputStream, readName(): hm
>>> KeyTab: load() entry length: 69; type: 18
>>> KeyTabInputStream, readName(): HADOOP.COM
>>> KeyTabInputStream, readName(): hive
>>> KeyTabInputStream, readName(): hm
>>> KeyTab: load() entry length: 53; type: 17
>>> KeyTabInputStream, readName(): HADOOP.COM
>>> KeyTabInputStream, readName(): hive
>>> KeyTabInputStream, readName(): hm
>>> KeyTab: load() entry length: 61; type: 16
>>> KeyTabInputStream, readName(): HADOOP.COM
>>> KeyTabInputStream, readName(): hive
>>> KeyTabInputStream, readName(): hm
>>> KeyTab: load() entry length: 53; type: 23
>>> KeyTabInputStream, readName(): HADOOP.COM
>>> KeyTabInputStream, readName(): hive
>>> KeyTabInputStream, readName(): hm
>>> KeyTab: load() entry length: 45; type: 8
>>> KeyTabInputStream, readName(): HADOOP.COM
>>> KeyTabInputStream, readName(): hive
>>> KeyTabInputStream, readName(): hm
>>> KeyTab: load() entry length: 45; type: 3
Added key: 3version: 1
Found unsupported keytype (8) for hive/hm@HADOOP.COM
Added key: 23version: 1
Added key: 16version: 1
Added key: 17version: 1
Found unsupported keytype (18) for hive/hm@HADOOP.COM
Ordering keys wrt default_tkt_enctypes list
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 17 16 23 1 3.
Added key: 3version: 1
Found unsupported keytype (8) for hive/hm@HADOOP.COM
Added key: 23version: 1
Added key: 16version: 1
Added key: 17version: 1
Found unsupported keytype (18) for hive/hm@HADOOP.COM
Ordering keys wrt default_tkt_enctypes list
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 17 16 23 1 3.
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 17 16 23 1 3.
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=hm UDP:88, timeout=30000, number of retries =3, #bytes=145
>>> KDCCommunication: kdc=hm UDP:88, timeout=30000,Attempt =1, #bytes=145
>>> KrbKdcReq send: #bytes read=598
>>> KdcAccessibility: remove hm
Added key: 3version: 1
Found unsupported keytype (8) for hive/hm@HADOOP.COM
Added key: 23version: 1
Added key: 16version: 1
Added key: 17version: 1
Found unsupported keytype (18) for hive/hm@HADOOP.COM
Ordering keys wrt default_tkt_enctypes list
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 17 16 23 1 3.
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbAsRep cons in KrbAsReq.getReply hive/hm
Added key: 3version: 1
Found unsupported keytype (8) for hive/hm@HADOOP.COM
Added key: 23version: 1
Added key: 16version: 1
Added key: 17version: 1
Found unsupported keytype (18) for hive/hm@HADOOP.COM
Ordering keys wrt default_tkt_enctypes list
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 17 16 23 1 3.
start connect hiveserver..
Found ticket for hive/hm@HADOOP.COM to go to krbtgt/HADOOP.COM@HADOOP.COM expiring on Wed
Jul 06 09:29:15 CST 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for hive/hm@HADOOP.COM to go to krbtgt/HADOOP.COM@HADOOP.COM expiring on Wed
Jul 06 09:29:15 CST 2016
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23 1 3.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbKdcReq send: kdc=hm UDP:88, timeout=30000, number of retries =3, #bytes=619
>>> KDCCommunication: kdc=hm UDP:88, timeout=30000,Attempt =1, #bytes=619
>>> KrbKdcReq send: #bytes read=116
>>> KdcAccessibility: remove hm
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
	 cTime is Wed Jul 04 22:58:32 CST 1984 457801112000
	 sTime is Tue Jul 05 09:29:15 CST 2016 1467682155000
	 suSec is 944361
	 error code is 37
	 error Message is Clock skew too great
	 realm is HADOOP.COM
	 sname is hive/hm
	 msgType is 30
KrbException: Clock skew too great (37) - PROCESS_TGS
	at sun.security.krb5.KrbTgsRep.<init>(Unknown Source)
	at sun.security.krb5.KrbTgsReq.getReply(Unknown Source)
	at sun.security.krb5.KrbTgsReq.sendAndGetCreds(Unknown Source)
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(Unknown Source)
	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(Unknown Source)
	at sun.security.krb5.Credentials.acquireServiceCreds(Unknown Source)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Unknown Source)
	at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
	at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown Source)
	at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Unknown Source)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
	at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
	at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at org.apache.hadoop.hive.ql.security.authorization.plugin.KerberosTest.main(KerberosTest.java:50)
Caused by: KrbException: Identifier doesn't match expected value (906)
	at sun.security.krb5.internal.KDCRep.init(Unknown Source)
	at sun.security.krb5.internal.TGSRep.init(Unknown Source)
	at sun.security.krb5.internal.TGSRep.<init>(Unknown Source)
	... 25 more
java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://hm:10000/default;principal=hive/hm@HADOOP.COM:
GSS initiate failed
	at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:231)
	at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at org.apache.hadoop.hive.ql.security.authorization.plugin.KerberosTest.main(KerberosTest.java:50)
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
	at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Unknown Source)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
	at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
	... 5 more

As if kerberos configuration is incorrect ....


At 2016-07-04 21:26:53, "Vivek Shrivastava" <vivshrivastava@gmail.com> wrote:
 

The renewal lifetime at client krb5.conf level does make any difference. The renewal time
period is defined at  kdc in kdc.conf. Client can not override it. The renewal is also a
property set at the principal level, both the settings ( renewal_lifetime, +renewal ) dictate
if a ticket can be renewed. I don't think your problem has anything to do with that. 


Seems something basic is missing in your environment. I would probably, run the same piece
of code in the unix environment and ensure that there is no error. Enabling Kerberos debugging
logging as suggested in the previous post will also help you compare the sequence of execution. 


On Mon, Jul 4, 2016 at 7:52 AM, Aviral Agarwal <aviral12028@gmail.com> wrote:


Hi,
Could you enable kerberos logs with 
    
    -Dsun.security.krb5.debug=true


and paste the output ?




On Mon, Jul 4, 2016 at 3:47 PM, Maria <linanmengxiaoai@126.com> wrote:

The qestion "kinit: Ticket expired while renewing credentials" has been solved. I can successfully
execute "kinit -R",

but the error “java.lang.RuntimeException: org.apache.thrift.transport.TTransportException:
Peer indicated failure: GSS initiate failed”

is still there..





At 2016-07-04 14:39:04, "Maria" <linanmengxiaoai@126.com> wrote:

>I saw a  mail named "HCatalog Security",His or her problem was similar to mine,and the
reply answer were:

>"This issue goes away after doing a kinit -R".

>

>So I did the same operation.while it is failed:

>kinit: Ticket expired while renewing credentials

>

>But in my /etc/krb5.conf, I have configed this item:

>renew_lifetime=7d

>

>So, Can anybody give me some suggestions, please? Thankyou.

>

>At 2016-07-04 11:32:30, "Maria" <linanmengxiaoai@126.com> wrote:

>>

>>

>>And  I can suucessfully access hiveserver2 from beeline.

>>

>>

>>I was so confused by this error"Peer indicated failure: GSS initiate failed".

>>

>> Can you anybody please help me? Any reply will be much appreciated.

>>

>>At 2016-07-04 11:26:53, "Maria" <linanmengxiaoai@126.com> wrote:

>>>Yup,my  hiveserver2 log errors are:

>>>

>>>ERROR [Hiveserver2-Handler-Pool: Thread-48]:server.TThreadPoolServer(TThreadPoolServer.java:run(296))
- error occurred during processing of message.

>>>java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer
indicated failure: GSS initiate failed

>>>    at org.apache.thrift.transport.TSaslServerTransport$FactorygetTransport(TSaslServerTransport.java:219)

>>>    at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)

>>>    at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)

>>>    at java.security.AccessController.doPrivileged(Native Method)

>>>    at javax.security.auth.Subject.doAs(Subject.java:356)

>>>    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1608)

>>>    at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)

>>>    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)

>>>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

>>>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

>>>    at java.lang.Thread.run(Thread.java:745)

>>>Caused by: org.apache.thrift.transport.TTransportException:Peer indicated failure:
GSS initiate failed

>>>    at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199)

>>>    at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)

>>>    at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)

>>>    at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)

>>>    at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)

>>> ... 10 more

>>>================================================

>>>As if the windows  hive JDBC client can communicate with the hiveserver2,isn't
it?

>>>

>>>while I checked everything I can :

>>>(1)in hiveserver2 node, I execute command "klist",the results are:

>>>Ticket cache: FILE:/tmp/krb5cc_0

>>>Default principal: hive/hm@HADOOP.COM

>>>

>>>Valid starting    Expires                     Service principal

>>>07/04/16 10:28:14    07/05/16 10:28:14     krbtgt/HADOOP.COM@HADOOP.COM

>>>                 renew until 07/04/16 10:28:14

>>>(2)in windows dos cmd,I execute command "klist",the results are:

>>>Ticket cache:API: 1

>>>Default principal: hive/hm@HADOOP.COM

>>>

>>>Valid starting    Expires                     Service principal

>>>07/04/16 10:24:32    07/05/16 10:24:32     krbtgt/HADOOP.COM@HADOOP.COM

>>>                 renew until 07/04/16 10:24:32

>>>

>>> Is there any thing else I have to add or set for hiveserver2?

>>>

>>>Thanks in advance.

>>>

>>>

>>>Maria.

>>>

>>>At 2016-07-03 04:39:31, "Vivek Shrivastava" <vivshrivastava@gmail.com> wrote:

>>>

>>>

>>>Please look at the hiveserver2 log, it will have better error information. You
can paste error from the logs if you need help. 

>>>

>>>

>>>Regards,

>>>

>>>

>>>Vivek

>>>

>>>

>>>On Sat, Jul 2, 2016 at 5:52 AM, Maria <linanmengxiaoai@126.com> wrote:

>>>

>>>

>>>

>>>Hi,all:

>>>

>>>     recently,I  attempted to access Kerberized hadoop cluster by launching
JAVA applications from Windows workstations. And I hava configured kerberos in my windows7,
and can successfully access hdfs50070. But when I launch JDBC from windows to connection remote
hiveserver,errors accured:

>>>

>>>java.sql.SQLException:could not open client transport with JDBC Uri:jdbc:hive2://hm:10000/default;principal=hive/hm@HADOOM.COM:
GSS initiate failed

>>>

>>>     at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:231)

>>>

>>>     at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)

>>>

>>>     at org.apache.hive.jdbc.HiveDriver.connection(HiveDriver.java:105)

>>>

>>>     at java.sql.DriverManager.getConnection(Unknown Source)

>>>

>>>     at java.sql.DriverManager.getConnection(Unknown Source)

>>>

>>>     at org.apache.hadoop.hive.ql.security.authorization.plugin.KerberosTest.main(KerberosTest.java:41)

>>>

>>>Caused by: org.apache.thrift.transport.TTransportException:GSS initiate failed

>>>

>>>     at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)

>>>

>>>     at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)

>>>

>>>     at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)

>>>

>>>     at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)

>>>

>>>     at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)

>>>

>>>     at java.security.AccessController.doPrivileged(Native Method)

>>>

>>>     at javax.security.auth.Subject.doAs(Unknow source)

>>>

>>>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

>>>

>>>     at  org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)

>>>

>>>     at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)

>>>

>>>... 5 more

>>>

>>>------------------------------------------------------------------------------

>>>

>>>below are my test codes:

>>>

>>>

>>>

>>>public static void main(String[] args) {

>>>

>>>    String principal = "hive/hm@HADOOM.COM";

>>>

>>>    String keytab = "E:\\Program Files (x86)\\java\\jre7\\lib\\security\\hive.keytab";

>>>

>>>    String url = "jdbc:hive2://hm:10000/default;principal=hive/hm@HADOOM.COM";

>>>

>>>

>>>

>>>    conf.addResource(new File("hdfs-site.xml").toURI().toURL());

>>>

>>>    conf.addResource(new File("core-site.xml").toURI().toURL());

>>>

>>>    conf.addResource(new File("yarn-site.xml").toURI().toURL());

>>>

>>>    conf.addResource(new File("hive-site.xml").toURI().toURL());

>>>

>>>

>>>

>>>    conf.set("hadoop.security.authentication", "Kerberos");

>>>

>>>    UserGroupInformation.setConfiguration(conf);

>>>

>>>    UserGroupInformation.loginUserFromKeytab(principal, keytab);

>>>

>>>

>>>

>>>    Class.forName("org.apache.hive.,jdbc.HiveDriver");

>>>

>>>    Connection conn =DriverManager.getConnection(url);

>>>

>>>

>>>

>>>    Statement stmt = conn.createStatement();

>>>

>>>    String sql = "select * from testkerberos";

>>>

>>>    ResultSet rs = stmt.executeQuery(sql);

>>>

>>>    while (rs.next()) {

>>>

>>>       system.out.println(rs.getString(1));

>>>

>>>    }

>>>

>>>}

>>>

>>>

>>>

>>>Does anyone had the same problem? Or know how to solve it ?

>>>

>>>

>>>

>>>Thanks in advance.

>>>

>>>

>>>

>>>Maria.

>>>

>>>

>>>





Mime
View raw message