accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyokwon Lee <hyokwon....@gmail.com>
Subject Re: Accumulo with Kerberos Error
Date Wed, 26 Feb 2014 19:09:41 GMT
Hi Charlie,

So from my understanding as of Accumulo 1.4.1  the need to manually kinit
before starting the process has been resolved.  I am currently running
1.4.3 and within the logs it shows that a new ticket is issued at the end
of expiration of the former ticket that was being held.   What version are
you running?

~Hokie


On Wed, Feb 26, 2014 at 1:57 PM, Ott, Charlie H.
<CHARLES.H.OTT@leidos.com>wrote:

>  Usually I manually issue the ticket to the accumulo process owners on
> all servers in the cluster.  Then using a bash script, you can create a
> cronjob that will renew the tickets.
>
>
>
> Run on master as accumulo user, ensuring passwordless ssh is enabled...
>
>
>
> #!/bin/bash
>
> ssh server2 "kinit -R"
>
> ssh server3 "kinit -R"
>
> ssh server3 "kinit -R"
>
>
>
>
>
> So if your tablet servers are running as user "accumulo", then I usually
> log in to the 'accumulo' account, perform a kinit, then start-all.sh
>
>
>
> I had a lot of issues with renewing the ticket before the expiration date,
> so after tweaking the principal properties, eventually I just set the
> ticket life for a year on that specific principal.
>
>
>
>
>
> *From:* user-return-3779-CHARLES.H.OTT=leidos.com@accumulo.apache.org[mailto:
> user-return-3779-CHARLES.H.OTT=leidos.com@accumulo.apache.org] *On Behalf
> Of *Hyokwon Lee
> *Sent:* Wednesday, February 26, 2014 1:40 PM
> *To:* user@accumulo.apache.org
> *Subject:* Re: Accumulo with Kerberos Error
>
>
>
> Hi All,
>
>
>
> I have an update to my issue.  Have turning on kerberos debug logging, I
> have found that a new kerberos ticket is initialized after the current
> ticket expires.  However table scans fail after the expiration of the
> original ticket that was issued at startup of accumulo services.  In the
> logs I see the following errors:
>
>
>
> When trying to scan a table:
>
> ERROR: java.lang.RuntimeException:
> org.apache.accumulo.core.client.impl.AccumuloServerException: Error on
> server accumulo.test.local:9997
>
>
>
> Numerous
>
> !METADATA -> FILE_READ -> /accumulo/tables/!0/root_tablet/xxxx
>
> Call to accumulo.test.local/127.0.0.1:8020 failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]
>
>
>
> and the log also keeps spamming:
>
>
>
> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/accumulo.test.local@TEST.LOCAL
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
>
>
> Has anyone seen this occur before?   I can simply recreate this issue by
> letting the accumulo services start up and expire and then just watching
> the logs and scanning the tables.
>
>
>
> Thanks,
>
>
>
> Hokie
>
>
>
> On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hyokwon.lee@gmail.com>
> wrote:
>
> Hi Sean,
>
>
>
> The Kerberos Tickets that are being used are not renewable.   Should they
> be?   I assume even if they are after their renewable time expires I will
> run into the same issue?
>
>
>
> Thanks,
>
>
>
> Hokie
>
>
>
> On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <busbey+lists@cloudera.com>
> wrote:
>
> Hi Hokie!
>
>
>
> Are the kerberos tickets you're getting renewable?
>
>
>
> -Sean
>
>
>
>
>
> On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hyokwon.lee@gmail.com>
> wrote:
>
> I am currently running into an issue and was hoping someone may have some
> insight to the problem.
>
>
>
> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed the
> following instructions in the README:
>
>
>
> "If you are running on top of hdfs with kerberos enabled, then you need to do
>
> some extra work. First, create an Accumulo principal
>
>
>
>   kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
>
>
>
> where <host.domain.name> is replaced by a fully qualified domain name. Export
>
> the principals to a keytab file. It is safer to create a unique keytab file for each
>
> server, but you can also glob them if you wish.
>
>
>
>   kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
>
>
>
> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
>
> the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
>
>
>
> In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
>
> and general.kerberos.principal, where the keytab setting is the absolute path
>
> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
>
> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
>
> _HOST in lieu of your individual host names.
>
>
>
>   <property>
>
>     <name>general.kerberos.keytab</name>
>
>     <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
>
>   </property>
>
>
>
>   <property>
>
>     <name>general.kerberos.principal</name>
>
>     <value>accumulo/_HOST@MYREALM</value>
>
>   </property>
>
>
>
> You can then start up Accumulo as you would with the accumulo user, and it will
>
> automatically handle the kerberos keys needed to access hdfs.
>
>
>
> Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
>
> You can resolve this by temporarily granting the accumulo user write access to the
>
> hdfs root directory, running init, and then revoking write permission in the root
>
> directory (be sure to maintain access to the /accumulo directory)."
>
>
>
>
>
>
>
> After doing so, got accumulo to come up and initially it states on start up that i authenticated
using accumulo/hostname.test.local@TEST.LOCAL.  For the next 24 hour it is happy and everything
works.   However after the 24 hour marker which is when the kerberos ticket expires, I start
seeing the following errors on all TServers:
>
>
>
>
>
>
>
> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
> [ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception:
GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
>
> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> And as far as I can tell this just retries and keeps failing.   I checked the accumulo.keytab
file and it is a glob so it has the entries for every server that Accumulo is on.   Also if
I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL it works
find and I get a valid ticket.  I also made sure everything in hdfs under "/accumulo" is owned
by accumulo so that doesn't seem to be the problem.  Also made sure after kiniting I can access
the directory path and all sub directories.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> So far the only thing that seems to fix my issue is if I bounce all accumulo services
and it is happy again.  Also until I bounce the accumulo services, I get error logs stating
it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.)
 Has anyone else seen this issue?  Did I miss a configuration somewhere possibly?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks,
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hokie
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> __________________________________________
> Hyokwon Lee
> hyokwon.lee@gmail.com
>
>
>
>
>
> --
>
> __________________________________________
> Hyokwon Lee
> hyokwon.lee@gmail.com
>



-- 
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com

Mime
View raw message