hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "donhoff_h" <165612...@qq.com>
Subject How does nodemanager spawned processes get principal from its driver?
Date Sat, 23 May 2015 13:11:50 GMT
Hi, all

I am not sure whether it is proper to publish my problem here. But just take a try.

I wrote a spark program to access HBase which is security enabled. The spark program is run
on Yarn(The problem should be related with nodemanager and it's why I try to publish the question
here). But my program could not pass HBase's authentication. After many times of tests, I
found that the cause reason is as following:

1. When my codes tried to authenticate with HBase, they use the Hadoop API "UserGroupInformation.getCurrentUser"
to get the "User" object to take authentication.

2. I read the source code of this API and find its logic is as following:
  AccessControlContext context = AccessController.getContext();
  Subject subject = Subject.getSubject(context);
  return subject != null && !subject.getPrincipals(User.class).isEmpty()?new UserGroupInformation(subject):getLoginUser();

3. I printed the context and subject objects, and found that the user information they contained
is just the linux system user "spark", not the kerberos principal "spark@BGDT.DEV.HRB". But
in fact , I did use the kinit command to set the principal on the program's driver node.

4. Since the context was the running process's context and the process was spawned by the
nodemanager of yarn, it means when the nodemanager spawned a new process for my spark program,
the new process did not get the principal from my program's driver.

So my problem is :
 How to set configurations to make sure the process spawned by nodemanager can get the principal
from its' driver node?
 Many Thanks!
View raw message