ZooKeeper<indexterm> - <primary>ZooKeeper</primary>

- SASL Authentication with ZooKeeper - Newer releases of Apache HBase (>= 0.92) will support connecting to a ZooKeeper Quorum - that supports SASL authentication (which is available in Zookeeper versions 3.4.0 or - later). - - This describes how to set up HBase to mutually authenticate with a ZooKeeper Quorum. - ZooKeeper/HBase mutual authentication (HBASE-2418) is required - as part of a complete secure HBase configuration (HBASE-3025). For - simplicity of explication, this section ignores additional configuration required (Secure HDFS - and Coprocessor configuration). It's recommended to begin with an HBase-managed Zookeeper - configuration (as opposed to a standalone Zookeeper quorum) for ease of learning. - -

- Operating System Prerequisites - - You need to have a working Kerberos KDC setup. For each $HOST that will - run a ZooKeeper server, you should have a principle zookeeper/$HOST. For each - such host, add a service key (using the kadmin or kadmin.local - tool's ktadd command) for zookeeper/$HOST and copy this file to - $HOST, and make it readable only to the user that will run zookeeper on - $HOST. Note the location of this file, which we will use below as - $PATH_TO_ZOOKEEPER_KEYTAB. - - Similarly, for each $HOST that will run an HBase server (master or - regionserver), you should have a principle: hbase/$HOST. For each host, add a - keytab file called hbase.keytab containing a service key for - hbase/$HOST, copy this file to $HOST, and make it readable only - to the user that will run an HBase service on $HOST. Note the location of this - file, which we will use below as $PATH_TO_HBASE_KEYTAB. - - Each user who will be an HBase client should also be given a Kerberos principal. This - principal should usually have a password assigned to it (as opposed to, as with the HBase - servers, a keytab file) which only this user knows. The client's principal's - maxrenewlife should be set so that it can be renewed enough so that the user - can complete their HBase client processes. For example, if a user runs a long-running HBase - client process that takes at most 3 days, we might create this user's principal within - kadmin with: addprinc -maxrenewlife 3days. The Zookeeper client - and server libraries manage their own ticket refreshment by running threads that wake up - periodically to do the refreshment. - - On each host that will run an HBase client (e.g. hbase shell), add the - following file to the HBase home directory's conf directory: - - -Client { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=false - useTicketCache=true; -}; - - - We'll refer to this JAAS configuration file as $CLIENT_CONF - below. -

- HBase-managed Zookeeper Configuration - - On each node that will run a zookeeper, a master, or a regionserver, create a JAAS - configuration file in the conf directory of the node's HBASE_HOME - directory that looks like the following: - - -Server { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=true - keyTab="$PATH_TO_ZOOKEEPER_KEYTAB" - storeKey=true - useTicketCache=false - principal="zookeeper/$HOST"; -}; -Client { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=true - useTicketCache=false - keyTab="$PATH_TO_HBASE_KEYTAB" - principal="hbase/$HOST"; -}; - - - where the $PATH_TO_HBASE_KEYTAB and - $PATH_TO_ZOOKEEPER_KEYTAB files are what you created above, and - $HOST is the hostname for that node. - - The Server section will be used by the Zookeeper quorum server, while the - Client section will be used by the HBase master and regionservers. The path - to this file should be substituted for the text $HBASE_SERVER_CONF in - the hbase-env.sh listing below. - - The path to this file should be substituted for the text - $CLIENT_CONF in the hbase-env.sh listing below. - - Modify your hbase-env.sh to include the following: - - -export HBASE_OPTS="-Djava.security.auth.login.config=$CLIENT_CONF" -export HBASE_MANAGES_ZK=true -export HBASE_ZOOKEEPER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" -export HBASE_MASTER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" -export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" - - - where $HBASE_SERVER_CONF and $CLIENT_CONF are - the full paths to the JAAS configuration files created above. - - Modify your hbase-site.xml on each node that will run zookeeper, - master or regionserver to contain: - - - - hbase.zookeeper.quorum - $ZK_NODES - - - hbase.cluster.distributed - true - - - hbase.zookeeper.property.authProvider.1 - org.apache.zookeeper.server.auth.SASLAuthenticationProvider - - - hbase.zookeeper.property.kerberos.removeHostFromPrincipal - true - - - hbase.zookeeper.property.kerberos.removeRealmFromPrincipal - true - - - ]]> - - where $ZK_NODES is the comma-separated list of hostnames of the Zookeeper - Quorum hosts. - - Start your hbase cluster by running one or more of the following set of commands on the - appropriate hosts: - - -bin/hbase zookeeper start -bin/hbase master start -bin/hbase regionserver start - - -

- -

- External Zookeeper Configuration - Add a JAAS configuration file that looks like: - -Client { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=true - useTicketCache=false - keyTab="$PATH_TO_HBASE_KEYTAB" - principal="hbase/$HOST"; -}; - - where the $PATH_TO_HBASE_KEYTAB is the keytab created above for - HBase services to run on this host, and $HOST is the hostname for that node. - Put this in the HBase home's configuration directory. We'll refer to this file's full - pathname as $HBASE_SERVER_CONF below. - - Modify your hbase-env.sh to include the following: - - -export HBASE_OPTS="-Djava.security.auth.login.config=$CLIENT_CONF" -export HBASE_MANAGES_ZK=false -export HBASE_MASTER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" -export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" - - - - Modify your hbase-site.xml on each node that will run a master or - regionserver to contain: - - - - hbase.zookeeper.quorum - $ZK_NODES - - - hbase.cluster.distributed - true - - - ]]> - - - where $ZK_NODES is the comma-separated list of hostnames of the Zookeeper - Quorum hosts. - - Add a zoo.cfg for each Zookeeper Quorum host containing: - -authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -kerberos.removeHostFromPrincipal=true -kerberos.removeRealmFromPrincipal=true - - Also on each of these hosts, create a JAAS configuration file containing: - -Server { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=true - keyTab="$PATH_TO_ZOOKEEPER_KEYTAB" - storeKey=true - useTicketCache=false - principal="zookeeper/$HOST"; -}; - - where $HOST is the hostname of each Quorum host. We will refer to the full - pathname of this file as $ZK_SERVER_CONF below. - - Start your Zookeepers on each Zookeeper Quorum host with: - -SERVER_JVMFLAGS="-Djava.security.auth.login.config=$ZK_SERVER_CONF" bin/zkServer start - - - Start your HBase cluster by running one or more of the following set of commands on the - appropriate nodes: - - -bin/hbase master start -bin/hbase regionserver start - - - -

- -

- Zookeeper Server Authentication Log Output - If the configuration above is successful, you should see something similar to the - following in your Zookeeper server logs: - -11/12/05 22:43:39 INFO zookeeper.Login: successfully logged in. -11/12/05 22:43:39 INFO server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 -11/12/05 22:43:39 INFO zookeeper.Login: TGT refresh thread started. -11/12/05 22:43:39 INFO zookeeper.Login: TGT valid starting at: Mon Dec 05 22:43:39 UTC 2011 -11/12/05 22:43:39 INFO zookeeper.Login: TGT expires: Tue Dec 06 22:43:39 UTC 2011 -11/12/05 22:43:39 INFO zookeeper.Login: TGT refresh sleeping until: Tue Dec 06 18:36:42 UTC 2011 -.. -11/12/05 22:43:59 INFO auth.SaslServerCallbackHandler: - Successfully authenticated client: authenticationID=hbase/ip-10-166-175-249.us-west-1.compute.internal@HADOOP.LOCALDOMAIN; - authorizationID=hbase/ip-10-166-175-249.us-west-1.compute.internal@HADOOP.LOCALDOMAIN. -11/12/05 22:43:59 INFO auth.SaslServerCallbackHandler: Setting authorizedID: hbase -11/12/05 22:43:59 INFO server.ZooKeeperServer: adding SASL authorization for authorizationID: hbase - - -

- -

- Zookeeper Client Authentication Log Output - On the Zookeeper client side (HBase master or regionserver), you should see something - similar to the following: - -11/12/05 22:43:59 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-166-175-249.us-west-1.compute.internal:2181 sessionTimeout=180000 watcher=master:60000 -11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.166.175.249:2181 -11/12/05 22:43:59 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 14851@ip-10-166-175-249 -11/12/05 22:43:59 INFO zookeeper.Login: successfully logged in. -11/12/05 22:43:59 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. -11/12/05 22:43:59 INFO zookeeper.Login: TGT refresh thread started. -11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-166-175-249.us-west-1.compute.internal/10.166.175.249:2181, initiating session -11/12/05 22:43:59 INFO zookeeper.Login: TGT valid starting at: Mon Dec 05 22:43:59 UTC 2011 -11/12/05 22:43:59 INFO zookeeper.Login: TGT expires: Tue Dec 06 22:43:59 UTC 2011 -11/12/05 22:43:59 INFO zookeeper.Login: TGT refresh sleeping until: Tue Dec 06 18:30:37 UTC 2011 -11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-166-175-249.us-west-1.compute.internal/10.166.175.249:2181, sessionid = 0x134106594320000, negotiated timeout = 180000 - -

- -

- Configuration from Scratch - - This has been tested on the current standard Amazon Linux AMI. First setup KDC and - principals as described above. Next checkout code and run a sanity check. - - -git clone git://git.apache.org/hbase.git -cd hbase -mvn clean test -Dtest=TestZooKeeperACL - - - Then configure HBase as described above. Manually edit target/cached_classpath.txt (see - below): - -bin/hbase zookeeper & -bin/hbase master & -bin/hbase regionserver & - -

- - -

- Future improvements - -

- Fix target/cached_classpath.txt - You must override the standard hadoop-core jar file from the - target/cached_classpath.txt file with the version containing the - HADOOP-7070 fix. You can use the following script to do this: - -echo `find ~/.m2 -name "*hadoop-core*7070*SNAPSHOT.jar"` ':' `cat target/cached_classpath.txt` | sed 's/ //g' > target/tmp.txt -mv target/tmp.txt target/cached_classpath.txt - -

- -

- Set JAAS configuration programmatically - - - This would avoid the need for a separate Hadoop jar that fixes HADOOP-7070. - -

- -

- Elimination of <code>kerberos.removeHostFromPrincipal</code> and - <code>kerberos.removeRealmFromPrincipal</code> - -

- -

- - -