From user-return-12837-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Thu Jun 11 11:28:36 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id E7F3D18065E for ; Thu, 11 Jun 2020 13:28:35 +0200 (CEST) Received: (qmail 32898 invoked by uid 500); 11 Jun 2020 11:28:34 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 32881 invoked by uid 99); 11 Jun 2020 11:28:34 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Jun 2020 11:28:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 73EAC181491 for ; Thu, 11 Jun 2020 11:28:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.255 X-Spam-Level: X-Spam-Status: No, score=0.255 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, KAM_SHORT=0.001, NORMAL_HTTP_TO_IP=0.001, NUMERIC_HTTP_ADDR=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id HVv0QL3y9IKH for ; Thu, 11 Jun 2020 11:28:30 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::b42; helo=mail-yb1-xb42.google.com; envelope-from=aparajita.1194@gmail.com; receiver= Received: from mail-yb1-xb42.google.com (mail-yb1-xb42.google.com [IPv6:2607:f8b0:4864:20::b42]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 37E357DD5E for ; Thu, 11 Jun 2020 11:28:29 +0000 (UTC) Received: by mail-yb1-xb42.google.com with SMTP id j202so2893181ybg.6 for ; Thu, 11 Jun 2020 04:28:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=BnOkAkGMVC7tujhUmrTXAvAUGzfF8ogB5s2vsh1vNms=; b=jyANL5xagzZJkE3kivOWA957TxHYXUtQyYyLKXHlO4FiTNZIHGaX0SWAzVbi/OIbXK VlujLbq8WUIxbi5Mpbt94U7DAZn7OkMS39ycwGWbLhTDceflogvnRdtedb58L10c12Kk tH/UndUbXQi2KR+b4eY8ZchNoVvh7fyvf5fvBO/p+lDTEkgkwMbjWaqdlFYEfUTq0+rC pkdokeEp6vPL6re6HK2QHe8OJIS/gHpCFJLBF4V/xydgyV3mVCKe8C14SGxTv+yV1zpN CRGFyR6CpE7D+HdUDFsiiOeJOGPGER/eF4aHJEBuQvGufk9sEfWZRMN73CK2PDJG8MGP hqew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=BnOkAkGMVC7tujhUmrTXAvAUGzfF8ogB5s2vsh1vNms=; b=epWwB4oEGfXS462wn3MI1WkVyUdZRxUFTCNYbHmhzI91+llxm/9TdX9MfcGWgM96Er xSjRqJAxFCFoYUIv3nbInR/WACQ6DIsleLH4aGjaGmgGVmgFH0jHNRX2UBWQzPUXmC8i nmCK/l5zp53+z9Hcvc3Q2wT2X/XO1iqjsrMkDfxKMVCOVtPrZp3MVq9zXE0JAL3m3biJ vooBUlUIquJ262FQ6HM068Gq8T4IVnxlwmcH0H3li6Jz7rVl1ij0t3meBlvnOjDZdQSw 7IJzaD/Fm+yLJROxGfmgFiCuzX0ZKUGba6JD4XVw2HItRt58Ma53hnoPCRcyhPdIZA2M A/xA== X-Gm-Message-State: AOAM530csUxwcNy43sRVqwz9GyDlI2bjELw2mOZwqVePW2I+na77dUlq Ch9gWzeIYVuAcyUzB70kcoCzM6qqeqDlX558uN04W+SP X-Google-Smtp-Source: ABdhPJzFzbYf8g+J9mX8lxSCV9MTee2BPP0NHgJw9aMwD9ivMafOta4sRX17W5i2udJKIflebUu+U4tnw16Y1+uNtNk= X-Received: by 2002:a25:ba03:: with SMTP id t3mr13123222ybg.425.1591874907273; Thu, 11 Jun 2020 04:28:27 -0700 (PDT) MIME-Version: 1.0 References: <93FEC11C-3ACD-4231-957F-228D53D46427@gmail.com> In-Reply-To: <93FEC11C-3ACD-4231-957F-228D53D46427@gmail.com> From: Aparajita Singh Date: Thu, 11 Jun 2020 16:58:15 +0530 Message-ID: Subject: Re: Zookeeper client fails during SASL authentication To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary="0000000000005fff1705a7cd42de" --0000000000005fff1705a7cd42de Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks Mate and Arpit, I'll check out your suggestions. Jorn, 1. Did you register the service principal correctly in your AD/KDC? 1. yes, the client and principals are registered in KDC. using kinit with keytab on the remote server was generating the TGT as expected. 2. If AD then did you make sure that the attribute for the user is activated to enable AES256 Kerberos auth? 1. i'm using KDC, AES256 was enabled by default. i changed the kdc.conf file to use only AES128 but zookeeper is still using AES256.= i'm looking to see if there is a config i have missed out somewhere durin= g zookeeper startup which is forcing it to use AES256 always. 3. Do you have unlimited crypto policies installed with your JDK? 1. yes, i verified this by checking if US_export_policy.jar and local_policy.jar files are present in $JAVA_HOME/jre/lib/security/unlimited 4. Is the keytab accessible to zk? 1. yes 5. Did you create keytab with AES256 encryption type? 1. yes, keytab had 2 entries for each principal corresponding to AES256 and AES128 On Thu, 11 Jun 2020 at 16:30, J=C3=B6rn Franke wrote= : > Kerberos can be quite a beast for any application. I managed to use > Kerberos authentication for Zookeeper a couple of times. Usually the erro= r > messages in Java are meaningless. > > Did you register the service principal correctly in your AD/KDC? > > If AD then did you make sure that the attribute for the user is activated > to enable AES256 Kerberos auth? > > Do you have unlimited crypto policies installed with your JDK? > > Is the keytab accessible to zk? > > Did you create keytab with AES256 encryption type? > > > > Am 10.06.2020 um 14:35 schrieb Aparajita Singh >: > > > > =EF=BB=BF > >> > >> > >> Hi, > >> > >> I am trying to migrate an unauthenticated zookeeper cluster to a > kerberos > >> authenticated one. For the time being SSL is disabled. I have configur= ed > >> the server and client as described below but when SASL is enabled I am > >> unable to retreive data using zookeeper shell client from the zookeepe= r > >> server. Could I get some help in understanding why this is failing? > >> > >> server.log snippet > >> > >> 2020-06-10 17:09:01,263 - INFO [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket > >> connection from /127.0.0.1:44994 > >> > >> 2020-06-10 17:09:01,264 - INFO [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from > / > >> 127.0.0.1:44994 > >> > >> 2020-06-10 17:09:01,265 - INFO [Thread-5:NIOServerCnxn@1007] - Closed > >> socket connection for client /127.0.0.1:44994 (no session established > for > >> client) > >> > >> 2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client > >> environment:zookeeper.version=3D3.4.6-169--1, built on 02/10/2016 05:4= 9 > GMT > >> > >> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client > >> environment:host.name=3Dstage-kdc-zk-ivy > >> > >> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client > >> environment:java.version=3D1.8.0_172 > >> > >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client > >> environment:java.vendor=3DOracle Corporation > >> > >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client > >> environment:java.home=3D/usr/lib/jvm/oracle-java8-jdk-amd64/jre > >> > >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client > >> > environment:java.class.path=3D/usr/hdp/2.4.0.0-169/zookeeper/bin/../build= /classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4= .0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-16= 9/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zook= eeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/= bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/= bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zooke= eper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../li= b/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j= -log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.= 1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr= /hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hd= p/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-sta= ble-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/= usr/hdp/ > 2.4.0. > 0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zook= eeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bi= n/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper= /bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../l= ib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-= plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-m= odel-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagno= stics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-ma= nager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.= 2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar= :/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.= 0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/b= in/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpco= re-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar= :/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/h= dp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169= /zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/= bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/.= ./lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/.= ./lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../= lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.= 8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.j= ar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.= 0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-= 169/zookeeper/zookeeper.jar:/usr/hdp/ > 2.4.0. > 0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zook= eeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-= api-1.6.1.jar:/usr/hdp/ > 2.4.0. > 0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zook= eeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore= -4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-= alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.= jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.= 0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/w= agon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.j= ar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169= /zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/com= mons-codec-1.6.jar:/usr/hdp/ > 2.4.0. > 0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib= /nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-conc= urrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.1= 7.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169= /zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib= /commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.= 2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp= /2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0= .0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper= /lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolatio= n-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.j= ar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr= /hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp= /2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2= .4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zoo= keeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wa= gon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-= manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.j= ar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/= zookeeper/* > >> > >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client > >> > environment:java.library.path=3D/usr/java/packages/lib/amd64:/usr/lib64:/= lib64:/lib:/usr/lib > >> > >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client > >> environment:java.io.tmpdir=3D/tmp > >> > >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client > >> environment:java.compiler=3D > >> > >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client > >> environment:os.name=3DLinux > >> > >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client > >> environment:os.arch=3Damd64 > >> > >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client > >> environment:os.version=3D4.9.0-9-amd64 > >> > >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client > >> environment:user.name=3Droot > >> > >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client > >> environment:user.home=3D/root > >> > >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client > >> environment:user.dir=3D/home/aparajita.singh > >> > >> 2020-06-10 17:09:26,653 - INFO [main:ZooKeeper@438] - Initiating > client > >> connection, connectString=3Dstage-kdc-zk-ivy sessionTimeout=3D30000 > >> watcher=3Dorg.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa > >> > >> 2020-06-10 17:09:26,752 - INFO > >> [main-SendThread(stage-kdc-zk-ivy:2181):Login@293] - successfully > logged > >> in. > >> > >> 2020-06-10 17:09:26,753 - INFO [Thread-0:Login$1@127] - TGT refresh > >> thread started. > >> > >> 2020-06-10 17:09:26,757 - INFO > >> [main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] - > >> Client will use GSSAPI as SASL mechanism. > >> > >> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@301] - TGT valid > starting > >> at: Wed Jun 10 15:17:21 IST 2020 > >> > >> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT expires: > >> Thu Jun 11 15:17:21 IST 2020 > >> > >> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login$1@181] - TGT refresh > >> sleeping until: Thu Jun 11 11:17:04 IST 2020 > >> > >> 2020-06-10 17:09:26,799 - INFO > >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] - > >> Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:218= 1 > . > >> Will attempt to SASL-authenticate using Login Context section 'Client' > >> > >> 2020-06-10 17:09:26,854 - INFO [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket > >> connection from /10.33.203.225:45018 > >> > >> 2020-06-10 17:09:26,854 - INFO > >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] - > >> Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181, > >> initiating session > >> > >> 2020-06-10 17:09:26,856 - INFO [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to > >> establish new session at /10.33.203.225:45018 > >> > >> 2020-06-10 17:09:26,859 - INFO [CommitProcessor:88:ZooKeeperServer@61= 7 > ] > >> - Established session 0x58729e0540980002 with negotiated timeout 30000 > for > >> client /10.33.203.225:45018 > >> > >> 2020-06-10 17:09:26,861 - INFO > >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] - > >> Session establishment complete on server stage-kdc-zk-ivy/ > >> 10.33.203.225:2181, sessionid =3D 0x58729e0540980002, negotiated timeo= ut > =3D > >> 30000 > >> > >> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL > >> authenticate: javax.security.sasl.SaslException: GSS initiate failed > >> [Caused by GSSException: Failure unspecified at GSS-API level (Mechani= sm > >> level: Invalid argument (400) - Cannot find key of appropriate type to > >> decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)] > >> > >> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975] - Closing client connection > due > >> to SASL authentication failure. > >> > >> 2020-06-10 17:09:27,007 - INFO [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection fo= r > >> client /10.33.203.225:45018 which had sessionid 0x58729e0540980002 > >> > >> 2020-06-10 17:09:27,008 - ERROR [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178] - Unexpected Exception: > >> > >> java.nio.channels.CancelledKeyException > >> > >> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) > >> > >> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) > >> > >> at > >> > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:1= 51) > >> > >> at > >> > org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java= :1081) > >> > >> at > >> > org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer= .java:936) > >> > >> at > >> > org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:= 373) > >> > >> at > >> > org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:= 200) > >> > >> at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244) > >> > >> at > >> > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory= .java:208) > >> > >> at java.lang.Thread.run(Thread.java:748) > >> > >> 2020-06-10 17:09:27,008 - INFO > >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] - > >> Unable to read additional data from server sessionid 0x58729e054098000= 2, > >> likely server has closed socket, closing socket connection and > attempting > >> reconnect > >> > >> 2020-06-10 17:09:27,008 - WARN [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346] - Exception causing close of > >> session 0x58729e0540980002 due to > java.nio.channels.CancelledKeyException > >> > >> 2020-06-10 17:10:01,317 - INFO [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket > >> connection from /127.0.0.1:45004 > >> > >> 2020-06-10 17:10:01,318 - INFO [NIOServerCxn.Factory: > >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from > / > >> 127.0.0.1:45004 > >> > >> > >> > >> zookeeper shell client output > >> > >> aparajita.singh@stage-kdc-zk-ivy:~$ sudo > >> /usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server > >> stage-kdc-zk-ivy get /test2 > >> > >> log4j:WARN Large window sizes are not allowed. > >> > >> log4j:WARN MaxIndex reduced to 13. > >> > >> Connecting to stage-kdc-zk-ivy > >> > >> Debug is true storeKey false useTicketCache true useKeyTab true > >> doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab = is > >> /etc/krb5.keytab refreshKrb5Config is false principal is > >> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false > >> useFirstPass is false storePass is false clearPass is false > >> > >> Acquire TGT from Cache > >> > >> Principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka > >> > >> null credentials from Ticket Cache > >> > >> principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka > >> > >> Will use keytab > >> > >> Commit Succeeded > >> > >> > >> > >> WATCHER:: > >> > >> > >> WatchedEvent state:SyncConnected type:None path:null > >> > >> > >> WATCHER:: > >> > >> > >> WatchedEvent state:Disconnected type:None path:null > >> > >> Exception in thread "main" > >> org.apache.zookeeper.KeeperException$ConnectionLossException: > >> KeeperErrorCode =3D ConnectionLoss for /test2 > >> > >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99= ) > >> > >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51= ) > >> > >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) > >> > >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) > >> > >> at > org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717) > >> > >> at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:59= 1) > >> > >> at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354) > >> > >> at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282) > >> > >> zoo.cfg > >> > >> #setACL=3DFalse > >> > >> autopurge.snapRetainCount=3D30 > >> > >> tickTime=3D2000 > >> > >> dataDir=3D/grid/1/var/lib/zookeeper > >> > >> zookeeper_jmx_port=3D9009 > >> > >> initLimit=3D100 > >> > >> syncLimit=3D5 > >> > >> autopurge.purgeInterval=3D24 > >> > >> clientPort=3D2181 > >> > >> globalOutstandingLimit=3D5000 > >> > >> maxClientCnxns=3D2000 > >> > >> server.99=3Dstage-kdc-zk-harley:2888:3888 > >> > >> server.88=3Dstage-kdc-zk-ivy:2888:3888 > >> > >> server.77=3Dstage-kdc-zk-2face:2888:3888 > >> > >> > >> > authProvider.1=3Dorg.apache.zookeeper.server.auth.SASLAuthenticationProvi= der > >> > >> requireClientAuthScheme=3Dsasl > >> > >> > >> quorum.auth.enableSasl=3Dtrue > >> > >> quorum.auth.learnerRequireSasl=3Dtrue > >> > >> quorum.auth.serverRequireSasl=3Dtrue > >> > >> > quorum.auth.kerberos.servicePrincipal=3Dhost/stage-kdc-zk-ivy@stage.fdp.k= afka > >> > >> quorum.cnxn.threads.size=3D20 > >> > >> > >> > >> java.env > >> > >> SERVER_JVMFLAGS=3D"${SERVER_JVMFLAGS} > >> -Djava.security.auth.login.config=3D/home/aparajita.singh/jaas/jaas.co= nf > >> > -Dzookeeper.authProvider.sasl=3Dorg.apache.zookeeper.server.auth.SASLAuth= enticationProvider > >> -Dsun.security.krb5.debug=3Dtrue" > >> > >> CLIENT_JVMFLAGS=3D"${CLIENT_JVMFLAGS} > >> -Djava.security.auth.login.config=3D/home/aparajita.singh/jaas/client.= conf > >> > -Dzookeeper.authProvider.sasl=3Dorg.apache.zookeeper.server.auth.SASLAuth= enticationProvider > >> -Dsun.security.krb5.debug=3Dtrue" > >> > >> > >> /home/aparajita.singh/jaas/jaas.conf > >> > >> // Zookeeper server authentication > >> > >> Server { > >> > >> com.sun.security.auth.module.Krb5LoginModule required > >> > >> useKeyTab=3Dtrue > >> > >> useTicketCache=3Dfalse > >> > >> //ticketCache=3D"/tmp/krb5cc_0" > >> > >> renewTicket=3Dtrue > >> > >> doNotPrompt=3Dtrue > >> > >> debug=3Dtrue > >> > >> keyTab=3D"/etc/krb5.keytab" > >> > >> serviceName=3D"host" > >> > >> principal=3D"host/stage-kdc-zk-ivy@stage.fdp.kafka"; > >> > >> }; > >> > >> > >> // Zookeeper quorum server authentication > >> > >> QuorumServer { > >> > >> com.sun.security.auth.module.Krb5LoginModule required > >> > >> useKeyTab=3Dtrue > >> > >> useTicketCache=3Dfalse > >> > >> //ticketCache=3D"/tmp/krb5cc_0" > >> > >> renewTicket=3Dtrue > >> > >> doNotPrompt=3Dtrue > >> > >> debug=3Dtrue > >> > >> keyTab=3D"/etc/krb5.keytab" > >> > >> serviceName=3D"host" > >> > >> principal=3D"host/stage-kdc-zk-ivy@stage.fdp.kafka"; > >> > >> }; > >> > >> > >> // Zookeeper learner authentication > >> > >> QuorumLearner { > >> > >> com.sun.security.auth.module.Krb5LoginModule required > >> > >> useKeyTab=3Dtrue > >> > >> useTicketCache=3Dfalse > >> > >> //ticketCache=3D"/tmp/krb5cc_0" > >> > >> renewTicket=3Dtrue > >> > >> doNotPrompt=3Dtrue > >> > >> debug=3Dtrue > >> > >> keyTab=3D"/etc/krb5.keytab" > >> > >> serviceName=3D"host" > >> > >> principal=3D"host/stage-kdc-zk-ivy@stage.fdp.kafka"; > >> > >> }; > >> > >> > >> > >> /home/aparajita.singh/jaas/client.conf > >> > >> // Zookeeper client authentication > >> > >> Client { > >> > >> com.sun.security.auth.module.Krb5LoginModule required > >> > >> useKeyTab=3Dtrue > >> > >> useTicketCache=3Dtrue > >> > >> ticketCache=3D"/tmp/krb5cc_0" > >> > >> renewTicket=3Dtrue > >> > >> doNotPrompt=3Dtrue > >> > >> debug=3Dtrue > >> > >> keyTab=3D"/etc/krb5.keytab" > >> > >> serviceName=3D"zookeeper" > >> > >> principal=3D"zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka"; > >> > >> }; > >> > >> > >> Using kinit command I am able to generate the TGT for both principals. > As > >> per the zookeeper server log, the TGT can be generated as expected. Th= e > >> keytab file is accessible to all system users for now. > >> > >> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit > >> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab > >> > >> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit > >> host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab > >> > >> > >> -- > >> Thanks, > >> Aparajita > >> > --=20 Thanks, Aparajita --0000000000005fff1705a7cd42de--