From common-dev-return-103083-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Wed Sep 11 06:22:04 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0A70218063F for ; Wed, 11 Sep 2019 08:22:03 +0200 (CEST) Received: (qmail 18450 invoked by uid 500); 11 Sep 2019 06:22:00 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 18423 invoked by uid 99); 11 Sep 2019 06:22:00 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Sep 2019 06:22:00 +0000 Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 38AE25A66; Wed, 11 Sep 2019 06:22:00 +0000 (UTC) Received: by mail-lf1-f46.google.com with SMTP id q11so526623lfc.11; Tue, 10 Sep 2019 23:22:00 -0700 (PDT) X-Gm-Message-State: APjAAAU6WK+AE+CiWNEGuaVxs6LSUJ0bVag1guXRkH8857YdvM7Xiaup +37j4K+GYWWMcCZ65l4oLTjhryTA1FWZIbMJhH4= X-Google-Smtp-Source: APXvYqxmEEtvkq7icBj3v/Zwl+cXoNmXEFS+FhRsizAEW9PyEwaNYYfCU8xS/ssHJWEqqndF17IDTbfycQdzQzTTWC4= X-Received: by 2002:ac2:5ec8:: with SMTP id d8mr23999909lfq.183.1568182919310; Tue, 10 Sep 2019 23:21:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Akira Ajisaka Date: Wed, 11 Sep 2019 15:21:39 +0900 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: RM and NM fails to start on Secure cluster with Java11 To: Prabhu Joseph Cc: Hadoop Common , yarn-dev Content-Type: multipart/alternative; boundary="000000000000d933a105924109cc" --000000000000d933a105924109cc Content-Type: text/plain; charset="UTF-8" Hi Prahbu, Is your principal allowed to use renewable tickets? If not, the client has to disable requests with renewable flag. Removing the following setting from krb5.conf worked for us. > renew_lifetime = 7d Details * https://bugs.openjdk.java.net/browse/JDK-8131051 * https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.security.jgss/share/classes/sun/security/krb5/KrbKdcRep.java#L83 Regards, Akira On Tue, Sep 10, 2019 at 5:46 PM Prabhu Joseph wrote: > RM and NM fails to start on Secure cluster with Java11 with below error > message " KrbException: Message stream modified (41)". Looks something > wrong with encryption types in Kerberos Configuration. Can someone give > pointers to debug the issue. > > > 2019-09-10 08:24:04,412 ERROR > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error > starting ResourceManager > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login > > at > > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:302) > > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > > at > > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1566) > > Caused by: org.apache.hadoop.security.KerberosAuthException: failure to > login: for principal: yarn/yarndocker-3@DOCKER.COM from keytab > /etc/security/keytabs/yarn.keytab javax.security.auth.login.LoginException: > Message stream modified (41) > > at > > org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008) > > at > > org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1376) > > at > > org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1156) > > at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:315) > > at > > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:1385) > > at > > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:300) > > ... 2 more > > Caused by: javax.security.auth.login.LoginException: Message stream > modified (41) > > at > > jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:781) > > at > > jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592) > > at > > java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726) > > at > > java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665) > > at > > java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663) > > at java.base/java.security.AccessController.doPrivileged(Native Method) > > at > > java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663) > > at > > java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574) > > at > > org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2087) > > at > > org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1998) > > ... 7 more > > Caused by: KrbException: Message stream modified (41) > > at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:83) > > at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:158) > > at > > java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121) > > at > > java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:295) > > at > > java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:371) > > at > > jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:753) > > ... 16 more > > > > > > > > [yarn@yarndocker-3 usr]$ cat /etc/krb5.conf > includedir /etc/krb5.conf.d/ > > [logging] > default = FILE:/var/log/krb5libs.log > kdc = FILE:/var/log/krb5kdc.log > admin_server = FILE:/var/log/kadmind.log > > [libdefaults] > default_tkt_enctypes=aes128-cts-hmac-sha1-96 > default_tgs_enctypes=aes128-cts-hmac-sha1-96 > dns_lookup_realm = false > ticket_lifetime = 24h > renew_lifetime = 7d > forwardable = true > rdns = false > default_realm = DOCKER.COM > default_ccache_name = /tmp/krb5cc_%{uid} > > [realms] > DOCKER.COM = { > kdc = yarndocker-3 > admin_server = yarndocker-3 > } > > > [yarn@yarndocker-3 usr]$ klist > Ticket cache: FILE:/tmp/krb5cc_1002 > Default principal: yarn/yarndocker-3@DOCKER.COM > > Valid starting Expires Service principal > 09/10/2019 08:12:24 09/11/2019 08:12:24 krbtgt/DOCKER.COM@DOCKER.COM > > > [root@yarndocker-3 logs]# cat /var/kerberos/krb5kdc/kdc.conf > [kdcdefaults] > kdc_ports = 88 > kdc_tcp_ports = 88 > > [realms] > EXAMPLE.COM = { > #master_key_type = aes256-cts > acl_file = /var/kerberos/krb5kdc/kadm5.acl > dict_file = /usr/share/dict/words > admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab > supported_enctypes = aes128-cts:normal des3-hmac-sha1:normal > arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal > des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal > } > > > > [root@yarndocker-3 logs]# java -version > > openjdk version "11.0.4" 2019-07-16 LTS > > OpenJDK Runtime Environment 18.9 (build 11.0.4+11-LTS) > > OpenJDK 64-Bit Server VM 18.9 (build 11.0.4+11-LTS, mixed mode, sharing) > --000000000000d933a105924109cc--