accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Srinivasan <>
Subject Re: Kerberos ticket renewal
Date Wed, 12 Jul 2017 16:01:43 GMT

So I've fired off a thread to perform the periodic
checkTGTAndReloginFromKeytab call which seems to be running, but the
connection still fails with GSS errors after precisely 10 hours.

While I am running 1.7.0, it seems the vendor included the
ACCUMULO-4069 patch, and immediately after the exception is thrown I
see a log entry "Performing ticket-cache-based Kerberos re-login".
However, it should be using a keytab - have turned up the logging to
11 and will leave running overnight...


On 11 July 2017 at 16:17, Josh Elser <> wrote:
> Nope, you've got it exactly right! That's the code I would've pointed you at
> to copy :)
> If/when you do get to long-running MR jobs, see the
> "general.delegation.token.*" configuration properties in this table[1]. I
> think the docs are citing that one delegation token is valid for 7 days, but
> it's been a long time since writing/testing that code.
> - Josh
> [1]
> On 7/11/17 1:25 AM, James Srinivasan wrote:
>> Thanks both. I can't (easily) upgrade beyond 1.7.0, but have raised a
>> support case with our Hadoop distribution vendor.
>> I'm not (yet) worried about expiration with MapReduce - for now I'll
>> try to keep such jobs to under 24h! Outside MR, sounds like I just
>> need to periodically call
>> UserGroupInformation.checkTGTAndReloginFromKeytab like
>> Or is the TGT associated with an Accumulo KerberosToken separate?
>> Thanks,
>> James
>> On 11 July 2017 at 02:59, Josh Elser <> wrote:
>>> No, you are (likely) not running into ACCUMULO-4069. What you've
>>> described sounds like your client's ticket expired. Accumulo does not
>>> spawn any ticket renewal on the behalf of clients.
>>> Hadoop's UGI code will automatically spawn a renewal thread when you
>>> log in using a ticket cache. This does not happen automatically when
>>> you use a keytab (I have no explanation as to why this is). This is
>>> the most likely cause of your error and something you need to correct
>>> in your application (spawn a thread to renew your application's
>>> ticket).
>>> If you are using MapReduce, you have yet another layer of indirection
>>> with DelegationTokens, but that's probably not what you're seeing (as
>>> DelegationTokens don't actually have a Kerberos TGT).
>>> On Mon, Jul 10, 2017 at 5:42 PM, Christopher <> wrote:
>>>> It certainly sounds like the same issue. I'd recommend upgrading to the
>>>> latest 1.7.3 (currently the latest 1.7 version) to include all the bugs
>>>> we've found and fixed in that release line.
>>>> On Mon, Jul 10, 2017 at 5:50 AM James Srinivasan
>>>> <> wrote:
>>>>> I'm using Accumulo 1.7.0 and finding that after some period of time
>>>>> (>8 hours, <3 days - happened over the weekend) my ingest fails
>>>>> errors regarding "Failed to find any Kerberos tgt". My guess is that
>>>>> the ticket from the keytab has expired, and needs to be renewed - from
>>>>> memory, I had seen a Kerberos tgt renewer thread running in my client,
>>>>> so assumed it happened automagically. Is that the case? Perhaps I am
>>>>> hitting this bug?
>>>>> Thanks,
>>>>> James

View raw message