hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daryn Sharp <da...@yahoo-inc.com>
Subject Re: Client mapred tries to renew a token with renewer specified as nobody
Date Mon, 09 Dec 2013 14:39:27 GMT
We encountered the same issue in yarn's RM so I made the RM recognize it's own tokens and renew
them regardless of the renewer.  In part because older versions of oozie hardcode the renewer
as "mrtoken".  I thought the change made it back into 1.x JT but I guess not.

I agree that the conversion to short name is occurring on the "wrong side of the fence". 
Auth_to_local is a function of the server, not the client.  Please file a jira.

Dfsclient also shouldn't be repeatedly retrying when the connection fails with InvalidToken.
 Hopefully the client didn't try more than 45 times (to clarify your "gazillion" from an earlier
message).   Some exceptions are transient, but an invalid token is game over because tokens
don't ever become valid again.  HA used to rely on InvalidToken to trigger retries in case
the standby is in the processing of catching up on edits, but that's been converted to a RetriableException
during a transition to active.   Please file another jira.

I would also expect  _HOST to expand correctly since a client may need to submit to multiple
JTs and not want to hassle with using different configs.  Would you like to file another jira?
:)

Daryn

On Dec 9, 2013, at 3:13 AM, Rainer Toebbicke <rtb@rtb-big-mac.cern.ch> wrote:

> 
> Le 5 déc. 2013 à 05:30, Vinod Kumar Vavilapalli a écrit :
> 
>> 
>> It is clearly mentioning that the renewer is wrong (renewer marked is 'nobody' but
mapred is trying to renew the token), you may want to check this.
>> 
>> Thanks,
>> +Vinod
>> 
>> On Dec 2, 2013, at 8:25 AM, Rainer Toebbicke wrote:
>> 
>>> 2013-12-02 15:57:08,541 ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:mapred/xxx.cern.ch@CERN.CH (auth:KERBEROS) cause:org.apache.hadoop.security.AccessControlException:
Client mapred tries to renew a token with renewer specified as nobody
>> 
> 
> Thanks, I had already guessed so far and finally the problem turned out a subtle bug
in the Cloudera "Configuring MRv1 Security" instructions: they recommend to specify mapreduce.jobtracker.kerberos.principal
as mapred/_HOST@YOUR-REALM.COM in mapred-site.xml. That won't work.
> 
> It confuses TokenCache.obtainTokensForNamenodesInternal() when it tries to obtain mapred's
HadoopKerberosName.getShortname(): it finds the untranslated _HOST which a correctly configured
hadoop.security.auth_to_local would not recognize as valid cluster node. That's where "nobody"
comes in. 
> 
> Solution: mapreduce.jobtracker.kerberos.principal should really be full a fully resolved
mapred/jobtracker-fqdn@YOUR-REALM.COM and thus would usually pass the hadoop.security.auth_to_local
rules.
> 
> One could argue that TokenCache.obtainTokensForNamenodesInternal() actually uses getShortname()
from the wrong side of the fence but the little change fixes the problem.
> 
> Thanks, Rainer


Mime
View raw message