accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyokwon Lee <hyokwon....@gmail.com>
Subject Re: Accumulo with Kerberos Error
Date Thu, 27 Feb 2014 15:30:14 GMT
Hi Charles and John,

So I made the tickets renewable and regenerated the keytabs for accumulo.
 The ticket life was set to 1 hour with the renew life set to 1 day.
 However after the hour is up, I get a different error:

Call to accumulo.test.local/127.0.0.1:8020 failed on local exception:
java.io.IOException: java.lang.IllegalStateException: This ticket is no
longer valid

immediately followed by:

javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]


I listed the ticket to make sure I had a valid ticket that was for 1
hour with a renew expiration of 1 day.  Then I went in and made sure
that the user running the service has a valid ticket, and just in case
on a few test added a cron job that renews the ticket before it
expires.   Either way I get the same error.


You mentioned that the system automatically renews the ticket when it
expires, and kerberos debug logging enabled I am seeing the following:


Found ticket for accumulo/accumulo.test.local@TEST.LOCAL to go to
krbtgt/TEST.LOCAL@TEST.LOCAL expiring on Thu Feb 27 07:14:20 PST 2014

Entered Krb5Context.initSecContext with state=STATE_NEW

Found ticket for accumulo/accumulo.test.local@TEST.LOCAL to go to
krbtgt/TEST.LOCAL@TEST.LOCAL expiring on Thu Feb 27 07:14:20 PST 2014

Found ticket for accumulo/accumulo.test.local@TEST.LOCAL to go to
hdfs/accumulo.test.local@TEST.LOCAL expiring on Thu Feb 27 07:14:20
PST 2014


There is no errors logged for the kerberos ticket creation however the
"This ticket is no longer valid " error leads me to believe that the
current ticket it had been using was destroyed.


Any thoughts?


Thanks,


Hokie



On Wed, Feb 26, 2014 at 3:11 PM, Hyokwon Lee <hyokwon.lee@gmail.com> wrote:

> Charles and John,
>
> Thanks for the help.   I am going to make the tickets renewable and give
> it a test.  I will let you guys know if it works.
>
> ~Hokie
>
>
> On Wed, Feb 26, 2014 at 2:43 PM, John Vines <vines@apache.org> wrote:
>
>> No, they need to be renewable. The system automatically renews them when
>> they expire.
>>
>>
>> On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hyokwon.lee@gmail.com>wrote:
>>
>>> Hi Sean,
>>>
>>> The Kerberos Tickets that are being used are not renewable.   Should
>>> they be?   I assume even if they are after their renewable time expires I
>>> will run into the same issue?
>>>
>>> Thanks,
>>>
>>> Hokie
>>>
>>>
>>> On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <busbey+lists@cloudera.com>wrote:
>>>
>>>> Hi Hokie!
>>>>
>>>> Are the kerberos tickets you're getting renewable?
>>>>
>>>> -Sean
>>>>
>>>>
>>>>
>>>> On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hyokwon.lee@gmail.com>wrote:
>>>>
>>>>> I am currently running into an issue and was hoping someone may have
>>>>> some insight to the problem.
>>>>>
>>>>> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed
>>>>> the following instructions in the README:
>>>>>
>>>>> "If you are running on top of hdfs with kerberos enabled, then you need
to do
>>>>> some extra work. First, create an Accumulo principal
>>>>>
>>>>>   kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
>>>>>
>>>>> where <host.domain.name> is replaced by a fully qualified domain
name. Export
>>>>> the principals to a keytab file. It is safer to create a unique keytab
file for each
>>>>> server, but you can also glob them if you wish.
>>>>>
>>>>>   kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
>>>>>
>>>>> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned
by
>>>>> the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
>>>>>
>>>>> In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
>>>>> and general.kerberos.principal, where the keytab setting is the absolute
path
>>>>> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is
set to
>>>>> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm.
You may use
>>>>> _HOST in lieu of your individual host names.
>>>>>
>>>>>   <property>
>>>>>     <name>general.kerberos.keytab</name>
>>>>>     <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
>>>>>   </property>
>>>>>
>>>>>   <property>
>>>>>     <name>general.kerberos.principal</name>
>>>>>     <value>accumulo/_HOST@MYREALM</value>
>>>>>   </property>
>>>>>
>>>>> You can then start up Accumulo as you would with the accumulo user, and
it will
>>>>> automatically handle the kerberos keys needed to access hdfs.
>>>>>
>>>>> Please Note: You may have issues initializing Accumulo while running
kerberos HDFS.
>>>>> You can resolve this by temporarily granting the accumulo user write
access to the
>>>>> hdfs root directory, running init, and then revoking write permission
in the root
>>>>> directory (be sure to maintain access to the /accumulo directory)."
>>>>>
>>>>>
>>>>> After doing so, got accumulo to come up and initially it states on start
up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL.  For the next 24 hour
it is happy and everything works.   However after the 24 hour marker which is when the kerberos
ticket expires, I start seeing the following errors on all TServers:
>>>>>
>>>>>
>>>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>>>
>>>>> [ipc.Client] WARN : Exception encountered while connecting to the server
: javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>>>
>>>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>>>
>>>>>
>>>>> And as far as I can tell this just retries and keeps failing.   I checked
the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo
is on.   Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL
it works find and I get a valid ticket.  I also made sure everything in hdfs under "/accumulo"
is owned by accumulo so that doesn't seem to be the problem.  Also made sure after kiniting
I can access the directory path and all sub directories.
>>>>>
>>>>>
>>>>> So far the only thing that seems to fix my issue is if I bounce all accumulo
services and it is happy again.  Also until I bounce the accumulo services, I get error logs
stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet,
etc.)  Has anyone else seen this issue?  Did I miss a configuration somewhere possibly?
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> Hokie
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> __________________________________________
>>> Hyokwon Lee
>>> hyokwon.lee@gmail.com
>>>
>>
>>
>
>
> --
> __________________________________________
> Hyokwon Lee
> hyokwon.lee@gmail.com
>



-- 
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com

Mime
View raw message