hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akira Ajisaka <aajis...@apache.org>
Subject Re: RM and NM fails to start on Secure cluster with Java11
Date Wed, 11 Sep 2019 06:21:39 GMT
Hi Prahbu,

Is your principal allowed to use renewable tickets? If not, the client has
to disable requests with renewable flag.
Removing the following setting from krb5.conf worked for us.

> renew_lifetime = 7d

Details
* https://bugs.openjdk.java.net/browse/JDK-8131051
*
https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.security.jgss/share/classes/sun/security/krb5/KrbKdcRep.java#L83

Regards,
Akira

On Tue, Sep 10, 2019 at 5:46 PM Prabhu Joseph <prabhujose.gates@gmail.com>
wrote:

> RM and NM fails to start on Secure cluster with Java11 with below error
> message " KrbException: Message stream modified (41)". Looks something
> wrong with encryption types in Kerberos Configuration. Can someone give
> pointers to debug the issue.
>
>
> 2019-09-10 08:24:04,412 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
> starting ResourceManager
>
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login
>
> at
>
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:302)
>
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>
> at
>
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1566)
>
> Caused by: org.apache.hadoop.security.KerberosAuthException: failure to
> login: for principal: yarn/yarndocker-3@DOCKER.COM from keytab
> /etc/security/keytabs/yarn.keytab javax.security.auth.login.LoginException:
> Message stream modified (41)
>
> at
>
> org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008)
>
> at
>
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1376)
>
> at
>
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1156)
>
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:315)
>
> at
>
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:1385)
>
> at
>
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:300)
>
> ... 2 more
>
> Caused by: javax.security.auth.login.LoginException: Message stream
> modified (41)
>
> at
>
> jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:781)
>
> at
>
> jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592)
>
> at
>
> java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
>
> at
>
> java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
>
> at
>
> java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
>
> at java.base/java.security.AccessController.doPrivileged(Native Method)
>
> at
>
> java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
>
> at
>
> java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
>
> at
>
> org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2087)
>
> at
>
> org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1998)
>
> ... 7 more
>
> Caused by: KrbException: Message stream modified (41)
>
> at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:83)
>
> at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:158)
>
> at
>
> java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
>
> at
>
> java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:295)
>
> at
>
> java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:371)
>
> at
>
> jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:753)
>
> ... 16 more
>
>
>
>
>
>
>
> [yarn@yarndocker-3 usr]$ cat /etc/krb5.conf
> includedir /etc/krb5.conf.d/
>
> [logging]
>  default = FILE:/var/log/krb5libs.log
>  kdc = FILE:/var/log/krb5kdc.log
>  admin_server = FILE:/var/log/kadmind.log
>
> [libdefaults]
> default_tkt_enctypes=aes128-cts-hmac-sha1-96
> default_tgs_enctypes=aes128-cts-hmac-sha1-96
>  dns_lookup_realm = false
>  ticket_lifetime = 24h
>  renew_lifetime = 7d
>  forwardable = true
>  rdns = false
>  default_realm = DOCKER.COM
>  default_ccache_name = /tmp/krb5cc_%{uid}
>
> [realms]
>  DOCKER.COM = {
>   kdc = yarndocker-3
>   admin_server = yarndocker-3
>  }
>
>
> [yarn@yarndocker-3 usr]$ klist
> Ticket cache: FILE:/tmp/krb5cc_1002
> Default principal: yarn/yarndocker-3@DOCKER.COM
>
> Valid starting       Expires              Service principal
> 09/10/2019 08:12:24  09/11/2019 08:12:24  krbtgt/DOCKER.COM@DOCKER.COM
>
>
> [root@yarndocker-3 logs]# cat /var/kerberos/krb5kdc/kdc.conf
> [kdcdefaults]
>  kdc_ports = 88
>  kdc_tcp_ports = 88
>
> [realms]
>  EXAMPLE.COM = {
>   #master_key_type = aes256-cts
>   acl_file = /var/kerberos/krb5kdc/kadm5.acl
>   dict_file = /usr/share/dict/words
>   admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
>   supported_enctypes = aes128-cts:normal des3-hmac-sha1:normal
> arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal
> des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
>  }
>
>
>
> [root@yarndocker-3 logs]# java -version
>
> openjdk version "11.0.4" 2019-07-16 LTS
>
> OpenJDK Runtime Environment 18.9 (build 11.0.4+11-LTS)
>
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.4+11-LTS, mixed mode, sharing)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message