hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Frain <ivan.fr...@gmail.com>
Subject AccessDeniedException on the regionserver with security enabled
Date Wed, 05 Sep 2012 09:45:42 GMT
Hi all,

My config:
 - Linux ubuntu 12.04
 - java 1.7.0_05 from oracle
 - hbase 0.94.1 compiled with "-Dhadoop.profile=2.0 -Psecurity -Prelease"
 - hadoop-2.0.0-alpha
 - zookeeper-3.4.3
 - mit kdc with REALM=HADOOP.LAN
 - all is running on a single ubuntu box which hostname is kdc.hadoop.lan

HDFS is working great with kerberos authentication.
Zookeeper is ok as well. I am able to connect "securerly" and play with
access control.

The problem happens when I launch the regionserver after having launched
the master. If I try to connect to hbase using the hbase shell, I obtain
the following error when I try to create a new table:

-----
hbase(main):003:0> create 'test_table','f'

ERROR: org.apache.hadoop.hbase.PleaseHoldException:
org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
------

Looking at the regionserver logs, I have the following warning:

------ hbase regionserver log ----
2012-09-05 11:32:43,297 WARN
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception
running postOpenDeployTasks; region=1028785192
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: AccessDeniedException: 1 time, servers with issues:
kdc.hadoop.lan:60020,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1601)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1377)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:916)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:772)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:747)
at org.apache.hadoop.hbase.catalog.MetaEditor.put(MetaEditor.java:99)
at
org.apache.hadoop.hbase.catalog.MetaEditor.putToCatalogTable(MetaEditor.java:89)
at
org.apache.hadoop.hbase.catalog.MetaEditor.updateLocation(MetaEditor.java:260)
at
org.apache.hadoop.hbase.catalog.MetaEditor.updateMetaLocation(MetaEditor.java:222)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1640)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:242)
-------

The master log file enters in a kind of infinite loop on the following log:

------- master log ------
2012-09-05 09:59:59,386 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x13995647afa0009 Successfully deleted unassigned node for
region 70236052 in expected state RS_ZK_REGION_OPENED
2012-09-05 09:59:59,393 INFO org.apache.hadoop.hbase.master.HMaster: -ROOT-
assigned=1, rit=false, location=kdc.hadoop.lan,60020,1346831988191
2012-09-05 09:59:59,394 INFO
org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
region -ROOT-,,0.70236052 that was online on
kdc.hadoop.lan,60020,1346831988191
2012-09-05 09:59:59,414 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6f318c24;
serverName=kdc.hadoop.lan,60020,1346831988191
2012-09-05 09:59:59,427 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6f318c24;
serverName=kdc.hadoop.lan,60020,1346831988191
2012-09-05 09:59:59,533 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6f318c24;
serverName=kdc.hadoop.lan,60020,1346831988191
2012-09-05 09:59:59,534 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6f318c24;
serverName=kdc.hadoop.lan,60020,1346831988191
2012-09-05 09:59:59,590 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6f318c24;
serverName=kdc.hadoop.lan,60020,1346831988191
------------------

Any Help on that would be much appreciated if anyone have an idea.

Thanks
Ivan

===========================
Appendix: the configuration files:

------- hbase-site.xml ----
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://kdc.hadoop.lan:8020/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>kdc.hadoop.lan</value>
  </property>
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.coprocessor.master.classes</name>
    <value>org.apache.hadoop.hbase.security.access.AccessController</value>
  </property>
  <property>
    <name>hbase.coprocessor.region.classes</name>
    <value>org.apache.hadoop.hbase.security.token.TokenProvider,
org.apache.hadoop.hbase.security.access.AccessController</value>
  </property>
  <property>
    <name>hbase.regionserver.kerberos.principal</name>
    <value>hbase/_HOST@HADOOP.LAN</value>
  </property>
  <property>
    <name>hbase.regionserver.keytab.file</name>
    <value>/home/ivan/hadoop/hbase/conf/hbase.keytab</value>
  </property>
  <property>
     <name>hbase.master.kerberos.principal</name>
     <value>hbase/_HOST@HADOOP.LAN</value>
  </property>
  <property>
     <name>hbase.master.keytab.file</name>
     <value>/home/ivan/hadoop/hbase/conf/hbase.keytab</value>
  </property>
</configuration>
-------

------ hbase-env.sh ------
# Set environment variables here.

# The java implementation to use.  Java 1.6 required.
export JAVA_HOME=/opt/java

# Extra Java CLASSPATH elements.  Optional.
export HBASE_CLASSPATH=/home/ivan/hadoop/conf/local-sec/

# Extra Java runtime options.
# Below are what we set by default.  May only work with SUN JVM.
# For more on why as well as other possible settings,
# see http://wiki.apache.org/hadoop/PerformanceTuning
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"


# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=false
export SECURITY_OPTS="-Djava.security.krb5.realm=HADOOP.LAN
-Djava.security.krb5.kdc=kdc.hadoop.lan
-Djava.security.auth.login.config=/home/ivan/hadoop/hbase/conf/jaas.conf"
export HBASE_OPTS="$HBASE_OPTS $SECURITY_OPTS"
export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $SECURITY_OPTS"
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $SECURITY_OPTS"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $SECURITY_OPTS"
----------

-------- jaas.conf ------
Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    useTicketCache=false
    doNotPrompt=true
    keyTab="/home/ivan/hadoop/hbase/conf/hbase.keytab"
    principal="hbase/kdc.hadoop.lan@HADOOP.LAN";
};
------

------ regionservers -----
kdc.hadoop.lan
------

-- 
Ivan Frain
11, route de Grenade
31530 Saint-Paul-sur-Save
mobile: +33 (0)6 52 52 47 07

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message