From user-return-12835-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Thu Jun 11 09:31:28 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 744DD18065E for ; Thu, 11 Jun 2020 11:31:28 +0200 (CEST) Received: (qmail 64766 invoked by uid 500); 11 Jun 2020 09:31:27 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 64754 invoked by uid 99); 11 Jun 2020 09:31:26 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Jun 2020 09:31:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id F135DC0248 for ; Thu, 11 Jun 2020 09:31:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.255 X-Spam-Level: X-Spam-Status: No, score=0.255 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, KAM_SHORT=0.001, NORMAL_HTTP_TO_IP=0.001, NUMERIC_HTTP_ADDR=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Xvr0C34TSOKW for ; Thu, 11 Jun 2020 09:31:22 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::e2f; helo=mail-vs1-xe2f.google.com; envelope-from=jain.arpit6@gmail.com; receiver= Received: from mail-vs1-xe2f.google.com (mail-vs1-xe2f.google.com [IPv6:2607:f8b0:4864:20::e2f]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 8F2FF7DD5E for ; Thu, 11 Jun 2020 09:31:21 +0000 (UTC) Received: by mail-vs1-xe2f.google.com with SMTP id l10so2995118vsr.10 for ; Thu, 11 Jun 2020 02:31:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=9pr4W/Pm87VpS+pGlPiW37Qk/lbWDJNUwATpkVfaH+M=; b=Q60Whht0xr99FfsKR+byhnY5K/H01JPlOExuV8tQe6l4wMOH/8hAQZsnARn9pUQCsy gx45PbmLeyVck0fkJOZcxQ0DvW5UGxqayX2OsBZaP+h+SL5BpzydhBB7fW8bPC6FGC7p mCMhzJsIDzz5gY0E1x/Nl2lNnYMQqGYJdnNBk1ZvfZEVdl/1mnCVut5csfViqqYwoX8r XRhFq5KlHC8ekZ0aWSdk1d332IIqfC8Krwnv8yVVd9L6kpFeNxy6j2RebmVF233HfYH+ U80qrAst7D2ZIu5+sqJCF+gWMvXLuNx8MbOyb6mX8XD4jWrYmiqMfKYlRMNMAMiXgH4R pg6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=9pr4W/Pm87VpS+pGlPiW37Qk/lbWDJNUwATpkVfaH+M=; b=ueZI7tO+wsZUTm1LnGlNVHD5w1USAjr9vB4FeWfRkooU4BNoWiKFh1RXECoGj3U6W2 J+/7IbvSUIfMeqxj1u+d+MsmLI4L5kPn4AzjnPHyFjtgqTBwXVZdknCwWEPFiWR/7ZPj sqhlTnaXG86qQvOs4uq6d9l3w44i3o27SNpiCOiWAoyvDWtKBhDbH9aN3ecx0ymzKywa ajLfSEbZydnB12bUNhV2RhWaJHbNuoIqgXU/E5ZISVGcBqbnNxWnfyvMVnvVMJaMd0ay yQ26lNZ3rFP/oJ1xs8T/Kc3GAfHMsRYRrs4XV2gPoOLPm34ld1dvVBADqhN7vCqxg4of bOiw== X-Gm-Message-State: AOAM533qhO/qKpHElTLINx1BKRgtbl564wB0rZvFctDplBTlaoxV5yRX iHIg3NENv5vLrNwmmqtPz5y8VWc/gW/BJZyyxs+HmhnKTXM= X-Google-Smtp-Source: ABdhPJxVI05P583JMA6jITo65kjcCZE8olGl80WGh8aSxXY14Yz6vezy6C1lbKTU0MQC7lUMSxraTylkqdofcXrKakI= X-Received: by 2002:a67:2d16:: with SMTP id t22mr6100225vst.160.1591867879735; Thu, 11 Jun 2020 02:31:19 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Arpit Jain Date: Thu, 11 Jun 2020 10:31:08 +0100 Message-ID: Subject: Re: Zookeeper client fails during SASL authentication To: UserZooKeeper Content-Type: multipart/alternative; boundary="00000000000080480805a7cb9fdc" --00000000000080480805a7cb9fdc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, I tried it a few months ago and managed to do it. I am not either an expert on this but managed to do the SASL authentication between ZK and client I ran the Kerberos server using this image https://hub.docker.com/r/gcavalcante8808/krb5-server/. Thanks On Thu, Jun 11, 2020 at 9:12 AM Szalay-Bek=C5=91 M=C3=A1t=C3=A9 wrote: > Hello Aparajita, > > After a quick glance on your configs and logs, I haven't found any proble= m > with your zookeeper configs. I am not sure if you know this page, using > these steps worked for me to setup a kerberized zookeeper: > https://github.com/ekoontz/zookeeper/wiki > I guess you are also familiar with our wiki: > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutua= l+authentication > > Based on your logs the problem is here: > 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory: > > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL > > authenticate: javax.security.sasl.SaslException: GSS initiate failed > > [Caused by GSSException: Failure unspecified at GSS-API level (Mechanis= m > > level: Invalid argument (400) - Cannot find key of appropriate type to > > decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)] > > > > This is a kerberos / jaas related issue, I don't think it is zookeeper > related. a few thing you might wish to check: > - make sure you have "Java Cryptography Extension (JCE) Unlimited Strengt= h > Jurisdiction Policy Files" installed (I think you need them for AES256?) > and your java security configs are OK > - run "klist -e -k /etc/krb5.keytab" to see if what encryptions you have > in the keytabs > - check if you have full export support in JCE by "java KeyLengthDetector= " > - Maybe you can try with different encryption types in kerberos configs / > during keytab generation. > - trying to use a different java version (latest JDK patches have some > known kerberos backward-incompatibilities) > > Unfortunately I am not a kerberos expert, so I don't know much about thes= e > issues, I just used google to find some hints :) > Maybe someone else in the community with deeper kerberos knowledge can he= lp > you more. > > Kind regards, > Mate > > On Thu, Jun 11, 2020 at 9:47 AM Aparajita Singh > wrote: > > > gentle reminder > > (unquoting the previous email) > > > > -- > > > > Hi, > > > > I am trying to migrate an unauthenticated zookeeper cluster to a kerber= os > > authenticated one. For the time being SSL is disabled. I have configure= d > > the server and client as described below but when SASL is enabled I am > > unable to retreive data using zookeeper shell client from the zookeeper > > server. Could I get some help in understanding why this is failing? > > > > > > *server.log snippet* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *2020-06-10 17:09:01,263 - INFO > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 > > ] - Accepted > socket > > connection from /127.0.0.1:44994 2020-06-10 > > 17:09:01,264 - INFO > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827 > > ] - Processing mntr > command > > from /127.0.0.1:44994 2020-06-10 17:09:01,265 - > > INFO [Thread-5:NIOServerCnxn@1007] - Closed socket connection for > client > > /127.0.0.1:44994 (no session established for > > client)2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client > > environment:zookeeper.version=3D3.4.6-169--1, built on 02/10/2016 05:49 > > GMT2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client > > environment:host.name =3Dstage-kdc-zk-ivy2020-06-10 > > 17:09:26,649 - INFO [main:Environment@100] - Client > > environment:java.version=3D1.8.0_1722020-06-10 17:09:26,651 - INFO > > [main:Environment@100] - Client environment:java.vendor=3DOracle > > Corporation2020-06-10 17:09:26,651 - INFO [main:Environment@100] - > Client > > environment:java.home=3D/usr/lib/jvm/oracle-java8-jdk-amd64/jre2020-06-= 10 > > 17:09:26,651 - INFO [main:Environment@100] - Client > > > > > environment:java.class.path=3D/usr/hdp/2.4.0.0-169/zookeeper/bin/../build= /classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4= .0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-16= 9/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zook= eeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/= bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/= bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zooke= eper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../li= b/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j= -log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.= 1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr= /hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hd= p/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-sta= ble-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/= usr/hdp/ > > 2.4.0. > > > 0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zook= eeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bi= n/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper= /bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../l= ib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-= plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-m= odel-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagno= stics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-ma= nager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.= 2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar= :/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.= 0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/b= in/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpco= re-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar= :/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/h= dp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169= /zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/= bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/.= ./lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/.= ./lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../= lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.= 8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.j= ar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.= 0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-= 169/zookeeper/zookeeper.jar:/usr/hdp/ > > 2.4.0. > > > 0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zook= eeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-= api-1.6.1.jar:/usr/hdp/ > > 2.4.0. > > > 0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zook= eeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore= -4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-= alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.= jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.= 0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/w= agon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.j= ar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169= /zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/com= mons-codec-1.6.jar:/usr/hdp/ > > 2.4.0. > > > 0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib= /nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-conc= urrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.1= 7.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169= /zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib= /commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.= 2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp= /2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0= .0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper= /lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolatio= n-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.j= ar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr= /hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp= /2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2= .4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zoo= keeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wa= gon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-= manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.j= ar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/= zookeeper/*2020-06-10 > > 17:09:26,651 - INFO [main:Environment@100] - Client > > > > > environment:java.library.path=3D/usr/java/packages/lib/amd64:/usr/lib64:/= lib64:/lib:/usr/lib2020-06-10 > > 17:09:26,651 - INFO [main:Environment@100] - Client > > environment:java.io.tmpdir=3D/tmp2020-06-10 17:09:26,651 - INFO > > [main:Environment@100] - Client > environment:java.compiler=3D2020-06-10 > > 17:09:26,651 - INFO [main:Environment@100] - Client environment:os.nam= e > > =3DLinux2020-06-10 17:09:26,652 - INFO > > [main:Environment@100] - Client environment:os.arch=3Damd642020-06-10 > > 17:09:26,652 - INFO [main:Environment@100] - Client > > environment:os.version=3D4.9.0-9-amd642020-06-10 17:09:26,652 - INFO > > [main:Environment@100] - Client environment:user.name > > =3Droot2020-06-10 17:09:26,652 - INFO > > [main:Environment@100] - Client environment:user.home=3D/root2020-06-1= 0 > > 17:09:26,652 - INFO [main:Environment@100] - Client > > environment:user.dir=3D/home/aparajita.singh2020-06-10 17:09:26,653 - I= NFO > > [main:ZooKeeper@438] - Initiating client connection, > > connectString=3Dstage-kdc-zk-ivy sessionTimeout=3D30000 > > watcher=3Dorg.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa2020-06-= 10 > > 17:09:26,752 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):Login@293] > - > > successfully logged in.2020-06-10 17:09:26,753 - INFO > > [Thread-0:Login$1@127] - TGT refresh thread started.2020-06-10 > > 17:09:26,757 - INFO > > [main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] - > > Client will use GSSAPI as SASL mechanism.2020-06-10 17:09:26,758 - INFO > > [Thread-0:Login@301] - TGT valid starting at: Wed Jun 10 > 15:17:21 > > IST 20202020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT > > expires: > > Thu Jun 11 15:17:21 IST 20202020-06-10 17:09:26,758 - > INFO > > [Thread-0:Login$1@181] - TGT refresh sleeping until: Thu Jun 11 > 11:17:04 > > IST 20202020-06-10 17:09:26,799 - INFO > > [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] - > > Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181 > > . Will attempt to SASL-authenticate using > Login > > Context section 'Client'2020-06-10 17:09:26,854 - INFO > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 > > ] - Accepted > socket > > connection from /10.33.203.225:45018 > >2020-06-10 > > 17:09:26,854 - INFO > > [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] - > > Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181 > > , initiating session2020-06-10 17:09:26,856 = - > > INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868 > > ] - Client attempting > to > > establish new session at /10.33.203.225:45018 > > 2020-06-10 17:09:26,859 - INFO > > [CommitProcessor:88:ZooKeeperServer@617] - Established session > > 0x58729e0540980002 with negotiated timeout 30000 for client > > /10.33.203.225:45018 2020-06-10 > 17:09:26,861 - > > INFO [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@127= 9 > ] > > - > > Session establishment complete on server > > stage-kdc-zk-ivy/10.33.203.225:2181 , > sessionid > > =3D 0x58729e0540980002, negotiated timeout =3D 300002020-06-10 17:09:27= ,007 - > > WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969 > > ] - Client failed to > SASL > > authenticate: javax.security.sasl.SaslException: GSS initiate failed > > [Caused by GSSException: Failure unspecified at GSS-API level (Mechanis= m > > level: Invalid argument (400) - Cannot find key of appropriate type to > > decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]2020-06-10 > 17:09:27,007 > > - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975 > > ] - Closing client > > connection due to SASL authentication failure.2020-06-10 17:09:27,007 - > > INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007 > > ] - Closed socket > > connection for client /10.33.203.225:45018 > > which had sessionid 0x58729e05409800022020-06-10 17:09:27,008 - ERROR > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178 > > ] - Unexpected Exception= : > > java.nio.channels.CancelledKeyExceptionat > > sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)at > > sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)at > > > > > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:1= 51)at > > > > > org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java= :1081)at > > > > > org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer= .java:936)at > > > > > org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:= 373)at > > > > > org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:= 200)at > > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)a= t > > > > > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory= .java:208)at > > java.lang.Thread.run(Thread.java:748)2020-06-10 17:09:27,008 - INFO > > [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] - > > Unable to read additional data from server sessionid 0x58729e0540980002= , > > likely server has closed socket, closing socket connection and attempti= ng > > reconnect2020-06-10 17:09:27,008 - WARN > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346 > > ] - Exception causing > close > > of session 0x58729e0540980002 due to > > java.nio.channels.CancelledKeyException2020-06-10 17:10:01,317 - INFO > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 > > ] - Accepted > socket > > connection from /127.0.0.1:45004 2020-06-10 > > 17:10:01,318 - INFO > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827 > > ] - Processing mntr > command > > from /127.0.0.1:45004 * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *zookeeper shell client outputaparajita.singh@stage-kdc-zk-ivy:~$ sudo > > /usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server > > stage-kdc-zk-ivy get /test2log4j:WARN Large window sizes are not > > allowed.log4j:WARN MaxIndex reduced to 13.Connecting to > > stage-kdc-zk-ivyDebug is true storeKey false useTicketCache true > useKeyTab > > true doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true > KeyTab > > is /etc/krb5.keytab refreshKrb5Config is false principal is > > zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false > > useFirstPass is false storePass is false clearPass is falseAcquire TGT > from > > CachePrincipal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafkanull > > credentials from Ticket Cacheprincipal is > > zookeeper/stage-kdc-zk-ivy@stage.fdp.kafkaWill use keytabCommit > Succeeded > > WATCHER::WatchedEvent state:SyncConnected type:None > > path:nullWATCHER::WatchedEvent state:Disconnected type:None > > path:nullException in thread "main" > > org.apache.zookeeper.KeeperException$ConnectionLossException: > > KeeperErrorCode =3D ConnectionLoss for /test2at > > org.apache.zookeeper.KeeperException.create(KeeperException.java:99)at > > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)at > > org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)at > > org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)at > > org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)= at > > org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)at > > org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)at > > > > > org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)zoo.cfg#se= tACL=3DFalseautopurge.snapRetainCount=3D30tickTime=3D2000dataDir=3D/grid/1/= var/lib/zookeeperzookeeper_jmx_port=3D9009initLimit=3D100syncLimit=3D5autop= urge.purgeInterval=3D24clientPort=3D2181globalOutstandingLimit=3D5000maxCli= entCnxns=3D2000server.99=3Dstage-kdc-zk-harley:2888:3888server.88=3Dstage-k= dc-zk-ivy:2888:3888server.77=3Dstage-kdc-zk-2face:2888:3888authProvider.1= =3Dorg.apache.zookeeper.server.auth.SASLAuthenticationProviderrequireClient= AuthScheme=3Dsaslquorum.auth.enableSasl=3Dtruequorum.auth.learnerRequireSas= l=3Dtruequorum.auth.serverRequireSasl=3Dtruequorum.auth.kerberos.servicePri= ncipal=3Dhost/stage-kdc-zk-ivy@stage.fdp.kafkaquorum.cnxn.threads.size > > =3D20* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *java.envSERVER_JVMFLAGS=3D"${SERVER_JVMFLAGS} > > -Djava.security.auth.login.config=3D/home/aparajita.singh/jaas/jaas.con= f > > > > > -Dzookeeper.authProvider.sasl=3Dorg.apache.zookeeper.server.auth.SASLAuth= enticationProvider > > -Dsun.security.krb5.debug=3Dtrue"CLIENT_JVMFLAGS=3D"${CLIENT_JVMFLAGS} > > -Djava.security.auth.login.config=3D/home/aparajita.singh/jaas/client.c= onf > > > > > -Dzookeeper.authProvider.sasl=3Dorg.apache.zookeeper.server.auth.SASLAuth= enticationProvider > > -Dsun.security.krb5.debug=3Dtrue"/home/aparajita.singh/jaas/jaas.conf// > > Zookeeper server authenticationServer { > > com.sun.security.auth.module.Krb5LoginModule required useKeyTab=3Dtr= ue > > useTicketCache=3Dfalse //ticketCache=3D"/tmp/krb5cc_0" renewTicke= t=3Dtrue > > doNotPrompt=3Dtrue debug=3Dtrue keyTab=3D"/etc/krb5.keytab" > > serviceName=3D"host" principal=3D"host/stage-kdc-zk-ivy@stage.fdp.ka= fka"; > > }; // Zookeeper quorum server authenticationQuorumServer { > > com.sun.security.auth.module.Krb5LoginModule required useKeyTab=3Dtr= ue > > useTicketCache=3Dfalse //ticketCache=3D"/tmp/krb5cc_0" renewTicke= t=3Dtrue > > doNotPrompt=3Dtrue debug=3Dtrue keyTab=3D"/etc/krb5.keytab" > > serviceName=3D"host" principal=3D"host/stage-kdc-zk-ivy@stage.fdp.ka= fka"; > > }; // Zookeeper learner authenticationQuorumLearner { > > com.sun.security.auth.module.Krb5LoginModule required useKeyTab=3Dtr= ue > > useTicketCache=3Dfalse //ticketCache=3D"/tmp/krb5cc_0" renewTicke= t=3Dtrue > > doNotPrompt=3Dtrue debug=3Dtrue keyTab=3D"/etc/krb5.keytab" > > serviceName=3D"host" principal=3D"host/stage-kdc-zk-ivy@stage.fdp.ka= fka"; > > }; /home/aparajita.singh/jaas/client.conf// Zookeeper client > > authenticationClient { com.sun.security.auth.module.Krb5LoginModule > > required useKeyTab=3Dtrue useTicketCache=3Dtrue > > ticketCache=3D"/tmp/krb5cc_0" renewTicket=3Dtrue doNotPrompt=3Dtr= ue > > debug=3Dtrue keyTab=3D"/etc/krb5.keytab" serviceName=3D"zookeeper= " > > principal=3D"zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka"; }; * > > Using kinit command I am able to generate the TGT for both principals. = As > > per the zookeeper server log, the TGT can be generated as expected. The > > keytab file is accessible to all system users for now. The below comman= ds > > don't give any output and the lack of error indicates that the ticket w= as > > generated successfully. klist command also shows the latest ticket > > generated as expected. > > > > *aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit > > zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab > > aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit > > host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab * > > > > > > Thanks, > > Aparajita > > > --00000000000080480805a7cb9fdc--