hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@apache.org>
Subject Re: HBase connection "expiration" on kerberized cluster
Date Thu, 03 Aug 2017 15:27:08 GMT
Hi schausson!

To get the ever-lasting connection you're looking for, you'll need to
provide a keytab that HBase can use to renew kerberos access over
time.

The AuthUtil class will take care of the configuration for you, see
the upcoming HBase 2 docs for an example:

http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/AuthUtil.html

The AuthUtil classes were marked as public API in HBase 1.3.0, but it
looks like we don't have those javadocs online yet.

The same tools are available in earlier HBase releases, but they're
not as polished and aren't marked as public yet:

http://hbase.apache.org/1.2/devapidocs/org/apache/hadoop/hbase/AuthUtil.html

It essentially still works, though there is a small risk that some
future critical issue might cause us to have to alter it.

Since you only need a single Connection per JVM, the start up /
teardown stuff given in the javadocs can go in application
initialization and cleanup. After the chore is running, you can just
create a Connection with the Configuration object you used and it
should work indefinitely.

On Thu, Aug 3, 2017 at 2:03 AM, schausson <schausson@softera.fr> wrote:
> Hi,
> I'm trying to setup web services that interact with my hadoop/hbase
> kerberized cluster.
> My application is deployed in a tomcat server and I would like to avoid
> recreating a new HBase connection each and every time I have to access
> HBase.
> Similarly, I want my application to be self sufficient, i.e I dont want to
> proceed with 'kinit' commands before starting up my tomcat server.
> Thus, I would like to implement a utility class in charge of managing login
> operation on the cluster and connection to hbase, but I'm struggling with
> kind of "ticket expiration" issues.
>
> First time my GetHbaseConnection() method is invoked, it properly connects
> to the cluster using provided keytab & principal (using
> UserGroupInformation.loginUserFromKeytab(user, keyTabPath) method), and
> create a brand new hbase connection
> (ConnectionFactory.createConnection(conf)) => perfect.
> By default, obtained ticket has a 10h lifetime (default value from
> /etc/krb5.conf file), so everything seems to work fine during first 10 hours
> period.
>
> Unfortunately, after this ticket has expired, my code fails with following
> exception :
> 17/08/01 07:40:52 http-nio-8443-exec-4 WARN  AbstractRpcClient:699 -
> Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 17/08/01 07:40:52 http-nio-8443-exec-4 ERROR AbstractRpcClient:709 - SASL
> authentication failed. The most likely cause is missing or invalid
> credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
>
> => I had to setup a dedicated thread that invoke
> UserGroupInformation.checkTGTAndReloginFromKeytab() method on a regular
> basis in order to refresh the ticket.
> Anyway, after a long time of inactivity (typically a whole night), when I
> try to invoke my web service, I can see following warnings in my tomcat logs
> :
>
> 17/08/03 08:25:28 hconnection-0x51b0ea6-shared--pool1-t51 WARN
> UserGroupInformation:1113 - Not attempting to re-login since the last
> re-login was attempted less than 600 seconds before.
> 17/08/03 08:25:29 hconnection-0x51b0ea6-shared--pool1-t51 WARN
> UserGroupInformation:1113 - Not attempting to re-login since the last
> re-login was attempted less than 600 seconds before.
> 17/08/03 08:25:30 hconnection-0x51b0ea6-shared--pool1-t51 WARN
> UserGroupInformation:1113 - Not attempting to re-login since the last
> re-login was attempted less than 600 seconds before.
> 17/08/03 08:25:31 hconnection-0x51b0ea6-shared--pool1-t51 WARN
> UserGroupInformation:1113 - Not attempting to re-login since the last
> re-login was attempted less than 600 seconds before.
> 17/08/03 08:25:35 hconnection-0x51b0ea6-shared--pool1-t51 WARN
> AbstractRpcClient:695 - Couldn't setup connection for myuser@mydomain.com to
> hbase/myserver.mydomain.com@mydomain.com
>
> ...And my call to the web service finally fails with
> SocketTimeoutException...
>
> To reproduce the issue quickly, I wrote a simple java application (outside
> of tomcat), removed the code that logs the user in the cluster to delegate
> this part to an external/manual kinit operation.
>         Proceed with a 'kinit' operation outside of my java application. This way I
> am able to get a "short-life" (1 minute) ticket using a custom krb5.conf
> file :
>
>         env KRB5_CONFIG=/local/home/myuser/mykrb5.conf kinit -kt
> /local/home/myuser/myuser.keytab myuser@mydomain.com
>
>         Then I execute my java standalone application that displays the name of one
> table in HBase on a regular basis (every 10 seconds). Note that I create a
> new HBase connection for every iteration, I dont try to reuse connection at
> the moment :
>
>         public static void main(String[] args) throws IOException,
> InterruptedException {
>                 System.setProperty( "sun.security.krb5.debug", "true");
>                 Configuration configuration = HBaseConfiguration.create();
>                 while (true) {
>                         Connection conn = ConnectionFactory.createConnection(configuration);
>                         Admin admin = conn.getAdmin();
>                         TableName[] tableNames = admin.listTableNames();
>                         System.out.println(tableNames[0].getNameWithNamespaceInclAsString());
>                         Thread.currentThread().sleep(10000);
>                 }
>         }
>
>         During 1 minute, it works perfectly, but then I face endless warnings and
> my code does not execute properly :
>
> 17/08/01 16:01:55 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 17/08/01 16:01:57 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 17/08/01 16:01:59 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 17/08/01 16:02:00 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 17/08/01 16:02:01 WARN ipc.AbstractRpcClient: Couldn't setup connection for
> myuser@mydomain.com to hbase/myserver.mydomain.com@mydomain.com
> ...
>
>
> I dont understand the how kerberos ticket expiration and hbase connection
> work together, does anyone could help on this topic ?
> In other words, I would like that my application connects to the cluster
> when it starts up, and create an hbase connection that I can keep "forever".
> Is it possible ? What did I miss ?
>
> Thanks for your help
>
>
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-connection-expiration-on-kerberized-cluster-tp4089493.html
> Sent from the HBase User mailing list archive at Nabble.com.

Mime
View raw message