zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aparajita Singh <aparajita.1...@gmail.com>
Subject Zookeeper client fails during SASL authentication
Date Wed, 10 Jun 2020 12:08:15 GMT
Hi,

I am trying to migrate an unauthenticated zookeeper cluster to a kerberos
authenticated one. For the time being SSL is disabled. I have configured
the server and client as described below but when SASL is enabled I am
unable to retreive data using zookeeper shell client from the zookeeper
server. Could I get some help in understanding why this is failing?

server.log snippet

2020-06-10 17:09:01,263 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /127.0.0.1:44994

2020-06-10 17:09:01,264 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
127.0.0.1:44994

2020-06-10 17:09:01,265 - INFO  [Thread-5:NIOServerCnxn@1007] - Closed
socket connection for client /127.0.0.1:44994 (no session established for
client)

2020-06-10 17:09:26,647 - INFO  [main:Environment@100] - Client
environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49 GMT

2020-06-10 17:09:26,649 - INFO  [main:Environment@100] - Client environment:
host.name=stage-kdc-zk-ivy

2020-06-10 17:09:26,649 - INFO  [main:Environment@100] - Client
environment:java.version=1.8.0_172

2020-06-10 17:09:26,651 - INFO  [main:Environment@100] - Client
environment:java.vendor=Oracle Corporation

2020-06-10 17:09:26,651 - INFO  [main:Environment@100] - Client
environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre

2020-06-10 17:09:26,651 - INFO  [main:Environment@100] - Client
environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*

2020-06-10 17:09:26,651 - INFO  [main:Environment@100] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2020-06-10 17:09:26,651 - INFO  [main:Environment@100] - Client
environment:java.io.tmpdir=/tmp

2020-06-10 17:09:26,651 - INFO  [main:Environment@100] - Client
environment:java.compiler=<NA>

2020-06-10 17:09:26,651 - INFO  [main:Environment@100] - Client environment:
os.name=Linux

2020-06-10 17:09:26,652 - INFO  [main:Environment@100] - Client
environment:os.arch=amd64

2020-06-10 17:09:26,652 - INFO  [main:Environment@100] - Client
environment:os.version=4.9.0-9-amd64

2020-06-10 17:09:26,652 - INFO  [main:Environment@100] - Client environment:
user.name=root

2020-06-10 17:09:26,652 - INFO  [main:Environment@100] - Client
environment:user.home=/root

2020-06-10 17:09:26,652 - INFO  [main:Environment@100] - Client
environment:user.dir=/home/aparajita.singh

2020-06-10 17:09:26,653 - INFO  [main:ZooKeeper@438] - Initiating client
connection, connectString=stage-kdc-zk-ivy sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa

2020-06-10 17:09:26,752 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):Login@293] - successfully logged in.

2020-06-10 17:09:26,753 - INFO  [Thread-0:Login$1@127] - TGT refresh thread
started.

2020-06-10 17:09:26,757 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] - Client
will use GSSAPI as SASL mechanism.

2020-06-10 17:09:26,758 - INFO  [Thread-0:Login@301] - TGT valid starting
at:        Wed Jun 10 15:17:21 IST 2020

2020-06-10 17:09:26,758 - INFO  [Thread-0:Login@302] - TGT expires:
          Thu Jun 11 15:17:21 IST 2020

2020-06-10 17:09:26,758 - INFO  [Thread-0:Login$1@181] - TGT refresh
sleeping until: Thu Jun 11 11:17:04 IST 2020

2020-06-10 17:09:26,799 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181.
Will attempt to SASL-authenticate using Login Context section 'Client'

2020-06-10 17:09:26,854 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /10.33.203.225:45018

2020-06-10 17:09:26,854 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] - Socket
connection established to stage-kdc-zk-ivy/10.33.203.225:2181, initiating
session

2020-06-10 17:09:26,856 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish
new session at /10.33.203.225:45018

2020-06-10 17:09:26,859 - INFO  [CommitProcessor:88:ZooKeeperServer@617] -
Established session 0x58729e0540980002 with negotiated timeout 30000 for
client /10.33.203.225:45018

2020-06-10 17:09:26,861 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] -
Session establishment complete on server stage-kdc-zk-ivy/10.33.203.225:2181,
sessionid = 0x58729e0540980002, negotiated timeout = 30000

2020-06-10 17:09:27,007 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL
authenticate: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
level: Invalid argument (400) - Cannot find key of appropriate type to
decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]

2020-06-10 17:09:27,007 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975] - Closing client connection due
to SASL authentication failure.

2020-06-10 17:09:27,007 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
client /10.33.203.225:45018 which had sessionid 0x58729e0540980002

2020-06-10 17:09:27,008 - ERROR [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178] - Unexpected Exception:

java.nio.channels.CancelledKeyException

at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)

at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)

at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)

at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)

at
org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)

at
org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)

at
org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)

at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)

at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

at java.lang.Thread.run(Thread.java:748)

2020-06-10 17:09:27,008 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
Unable to read additional data from server sessionid 0x58729e0540980002,
likely server has closed socket, closing socket connection and attempting
reconnect

2020-06-10 17:09:27,008 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346] - Exception causing close of
session 0x58729e0540980002 due to java.nio.channels.CancelledKeyException

2020-06-10 17:10:01,317 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /127.0.0.1:45004

2020-06-10 17:10:01,318 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
127.0.0.1:45004



zookeeper shell client output

aparajita.singh@stage-kdc-zk-ivy:~$ sudo
/usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
stage-kdc-zk-ivy get /test2

log4j:WARN Large window sizes are not allowed.

log4j:WARN MaxIndex reduced to 13.

Connecting to stage-kdc-zk-ivy

Debug is  true storeKey false useTicketCache true useKeyTab true
doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab is
/etc/krb5.keytab refreshKrb5Config is false principal is
zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false
useFirstPass is false storePass is false clearPass is false

Acquire TGT from Cache

Principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka

null credentials from Ticket Cache

principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka

Will use keytab

Commit Succeeded



WATCHER::


WatchedEvent state:SyncConnected type:None path:null


WATCHER::


WatchedEvent state:Disconnected type:None path:null

Exception in thread "main"
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /test2

at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)

at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)

at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)

at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)

at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)

at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)

at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)

zoo.cfg

#setACL=False

autopurge.snapRetainCount=30

tickTime=2000

dataDir=/grid/1/var/lib/zookeeper

zookeeper_jmx_port=9009

initLimit=100

syncLimit=5

autopurge.purgeInterval=24

clientPort=2181

globalOutstandingLimit=5000

maxClientCnxns=2000

server.99=stage-kdc-zk-harley:2888:3888

server.88=stage-kdc-zk-ivy:2888:3888

server.77=stage-kdc-zk-2face:2888:3888


authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider

requireClientAuthScheme=sasl


quorum.auth.enableSasl=true

quorum.auth.learnerRequireSasl=true

quorum.auth.serverRequireSasl=true

quorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-ivy@stage.fdp.kafka

quorum.cnxn.threads.size=20



java.env

SERVER_JVMFLAGS="${SERVER_JVMFLAGS}
-Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
-Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-Dsun.security.krb5.debug=true"

CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
-Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
-Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-Dsun.security.krb5.debug=true"


/home/aparajita.singh/jaas/jaas.conf

// Zookeeper server authentication

Server {

    com.sun.security.auth.module.Krb5LoginModule required

    useKeyTab=true

    useTicketCache=false

    //ticketCache="/tmp/krb5cc_0"

    renewTicket=true

    doNotPrompt=true

    debug=true

    keyTab="/etc/krb5.keytab"

    serviceName="host"

    principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";

    };


// Zookeeper quorum server authentication

QuorumServer {

    com.sun.security.auth.module.Krb5LoginModule required

    useKeyTab=true

    useTicketCache=false

    //ticketCache="/tmp/krb5cc_0"

    renewTicket=true

    doNotPrompt=true

    debug=true

    keyTab="/etc/krb5.keytab"

    serviceName="host"

    principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";

    };


// Zookeeper learner authentication

QuorumLearner {

    com.sun.security.auth.module.Krb5LoginModule required

    useKeyTab=true

    useTicketCache=false

    //ticketCache="/tmp/krb5cc_0"

    renewTicket=true

    doNotPrompt=true

    debug=true

    keyTab="/etc/krb5.keytab"

    serviceName="host"

    principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";

    };



/home/aparajita.singh/jaas/client.conf

// Zookeeper client authentication

Client {

    com.sun.security.auth.module.Krb5LoginModule required

    useKeyTab=true

    useTicketCache=true

    ticketCache="/tmp/krb5cc_0"

    renewTicket=true

    doNotPrompt=true

    debug=true

    keyTab="/etc/krb5.keytab"

    serviceName="zookeeper"

    principal="zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka";

    };


Using kinit command I am able to generate the TGT for both principals. As
per the zookeeper server log, the TGT can be generated as expected. The
keytab file is accessible to all system users for now.

aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab

aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab


-- 
Thanks,
Aparajita

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message