accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ott, Charlie H." <CHARLES.H....@leidos.com>
Subject RE: Accumulo with Kerberos Error
Date Thu, 27 Feb 2014 15:46:13 GMT
My only thoughts are that this is why I ended up setting my ticket life to a year for the accumulo
principal.

I actually had tickets renewing daily for about a week or so, then it seemed like out of nowhere,
they would occasionally be expired as if the renewal had not occurred.  Hopefully someone
can give you some better feedback than that.

Charles

From: user-return-3791-CHARLES.H.OTT=leidos.com@accumulo.apache.org [mailto:user-return-3791-CHARLES.H.OTT=leidos.com@accumulo.apache.org]
On Behalf Of Hyokwon Lee
Sent: Thursday, February 27, 2014 10:30 AM
To: user@accumulo.apache.org
Subject: Re: Accumulo with Kerberos Error

Hi Charles and John,

So I made the tickets renewable and regenerated the keytabs for accumulo.  The ticket life
was set to 1 hour with the renew life set to 1 day.  However after the hour is up, I get a
different error:

Call to accumulo.test.local/127.0.0.1:8020<http://127.0.0.1:8020> failed on local exception:
java.io.IOException: java.lang.IllegalStateException: This ticket is no longer valid

immediately followed by:


javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
provided (Mechanism level: Failed to find any Kerberos tgt)]



I listed the ticket to make sure I had a valid ticket that was for 1 hour with a renew expiration
of 1 day.  Then I went in and made sure that the user running the service has a valid ticket,
and just in case on a few test added a cron job that renews the ticket before it expires.
  Either way I get the same error.



You mentioned that the system automatically renews the ticket when it expires, and kerberos
debug logging enabled I am seeing the following:



Found ticket for accumulo/accumulo.test.local@TEST.LOCAL<mailto:accumulo/accumulo.test.local@TEST.LOCAL>
to go to krbtgt/TEST.LOCAL@TEST.LOCAL<mailto:krbtgt/TEST.LOCAL@TEST.LOCAL> expiring
on Thu Feb 27 07:14:20 PST 2014

Entered Krb5Context.initSecContext with state=STATE_NEW



Found ticket for accumulo/accumulo.test.local@TEST.LOCAL<mailto:accumulo/accumulo.test.local@TEST.LOCAL>
to go to krbtgt/TEST.LOCAL@TEST.LOCAL<mailto:krbtgt/TEST.LOCAL@TEST.LOCAL> expiring
on Thu Feb 27 07:14:20 PST 2014

Found ticket for accumulo/accumulo.test.local@TEST.LOCAL<mailto:accumulo/accumulo.test.local@TEST.LOCAL>
to go to hdfs/accumulo.test.local@TEST.LOCAL<mailto:hdfs/accumulo.test.local@TEST.LOCAL>
expiring on Thu Feb 27 07:14:20 PST 2014



There is no errors logged for the kerberos ticket creation however the "This ticket is no
longer valid " error leads me to believe that the current ticket it had been using was destroyed.



Any thoughts?



Thanks,





Hokie

On Wed, Feb 26, 2014 at 3:11 PM, Hyokwon Lee <hyokwon.lee@gmail.com<mailto:hyokwon.lee@gmail.com>>
wrote:
Charles and John,

Thanks for the help.   I am going to make the tickets renewable and give it a test.  I will
let you guys know if it works.

~Hokie

On Wed, Feb 26, 2014 at 2:43 PM, John Vines <vines@apache.org<mailto:vines@apache.org>>
wrote:
No, they need to be renewable. The system automatically renews them when they expire.

On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hyokwon.lee@gmail.com<mailto:hyokwon.lee@gmail.com>>
wrote:
Hi Sean,

The Kerberos Tickets that are being used are not renewable.   Should they be?   I assume even
if they are after their renewable time expires I will run into the same issue?

Thanks,

Hokie

On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <busbey+lists@cloudera.com<mailto:busbey+lists@cloudera.com>>
wrote:
Hi Hokie!

Are the kerberos tickets you're getting renewable?

-Sean


On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hyokwon.lee@gmail.com<mailto:hyokwon.lee@gmail.com>>
wrote:
I am currently running into an issue and was hoping someone may have some insight to the problem.

Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed the following instructions
in the README:


"If you are running on top of hdfs with kerberos enabled, then you need to do

some extra work. First, create an Accumulo principal



  kadmin.local -q "addprinc -randkey accumulo/<host.domain.name<http://host.domain.name>>"



where <host.domain.name<http://host.domain.name>> is replaced by a fully qualified
domain name. Export

the principals to a keytab file. It is safer to create a unique keytab file for each

server, but you can also glob them if you wish.



  kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"



Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by

the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh



In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab

and general.kerberos.principal, where the keytab setting is the absolute path

to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to

accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use

_HOST in lieu of your individual host names.



  <property>

    <name>general.kerberos.keytab</name>

    <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>

  </property>



  <property>

    <name>general.kerberos.principal</name>

    <value>accumulo/_HOST@MYREALM</value>

  </property>



You can then start up Accumulo as you would with the accumulo user, and it will

automatically handle the kerberos keys needed to access hdfs.



Please Note: You may have issues initializing Accumulo while running kerberos HDFS.

You can resolve this by temporarily granting the accumulo user write access to the

hdfs root directory, running init, and then revoking write permission in the root

directory (be sure to maintain access to the /accumulo directory)."







After doing so, got accumulo to come up and initially it states on start up that i authenticated
using accumulo/hostname.test.local@TEST.LOCAL<mailto:accumulo/hostname.test.local@TEST.LOCAL>.
 For the next 24 hour it is happy and everything works.   However after the 24 hour marker
which is when the kerberos ticket expires, I start seeing the following errors on all TServers:







[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

[ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception:
GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]

[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]





















And as far as I can tell this just retries and keeps failing.   I checked the accumulo.keytab
file and it is a glob so it has the entries for every server that Accumulo is on.   Also if
I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL<mailto:accumulo/tserver1.test.local@TEST.LOCAL>
it works find and I get a valid ticket.  I also made sure everything in hdfs under "/accumulo"
is owned by accumulo so that doesn't seem to be the problem.  Also made sure after kiniting
I can access the directory path and all sub directories.





















So far the only thing that seems to fix my issue is if I bounce all accumulo services and
it is happy again.  Also until I bounce the accumulo services, I get error logs stating it
cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.)
 Has anyone else seen this issue?  Did I miss a configuration somewhere possibly?





















Thanks,





















Hokie





















--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<mailto:hyokwon.lee@gmail.com>




--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<mailto:hyokwon.lee@gmail.com>



--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<mailto:hyokwon.lee@gmail.com>

Mime
View raw message