reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Lowe <>
Subject Re: Two AMs in one YARN container?
Date Thu, 16 Mar 2017 21:34:31 GMT
The doAs method in UserGroupInformation is what you want when dealing with multiple UGIs. 
It determines what UGI instance the code within the doAs scope gets when that code tries to
lookup the current user.
Each AM is designed to run in a separate JVM, so each has some main()-like entry point that
does everything to setup the AM.  Theoretically all you need to do is create two, separate
UGIs then use each instance to perform a doAs wrapping the invocation of the corresponding
AM's entry point.  After that, everything that AM does will get the UGI of the doAs invocation
as the current user.  Since the AMs are running in separate doAs instances they will get
separate UGIs for the current user and thus separate credentials.

    On Thursday, March 16, 2017 4:03 PM, Sergiy Matusevych <>

 Hi Jason,

Thanks a lot for your help again! Having two separate UserGroupInformation instances is exactly
what I had in mind. What I do not understand, though, is how to make sure that our second
call to .regsiterApplicationMaster() will pick the right UserGroupInformation object. I would
love to find a way that does not involve any changes to the YARN client, but if we have to
patch it, of course, I agree that we need to have a generic yet minimally invasive solution.
Thank you!Sergiy.

On Thu, Mar 16, 2017 at 8:03 AM, Jason Lowe <> wrote:
> I believe a cleaner way to solve this problem is to create two, _separate_ UserGroupInformation
objects and wrap each AM instances in a UGI doAs so they aren't trying to share the same credentials. 
This is one example of a token bleeding over and causing problems. I suspect trying to fix
these one-by-one as they pop up is going to be frustrating compared to just ensuring the credentials
remain separate as if they really were running in separate JVMs.
> Adding Daryn who knows a lot more about the UGI stuff so he can correct any misunderstandings
on my part.
> Jason
> On Wednesday, March 15, 2017 1:11 AM, Sergiy Matusevych <>
> Hi YARN developers,
> I have an interesting problem that I think is related to YARN Java client.
> I am trying to launch *two* application masters in one container. To be
> more specific, I am starting a Spark job on YARN, and launch an Apache REEF
> Unmanaged AM from the Spark Driver.
> Technically, YARN Resource Manager should not care which process each AM
> runs in. However, there is a problem with the YARN Java client
> implementation: there is a global UserGroupInformation object that holds
> the user credentials of the current RM session. This data structure is
> shared by all AMs, and when REEF application tries to register the second
> (unmanaged) AM, the client library presents to YARN RM all credentials,
> including the security token of the first (managed) AM. YARN rejects such
> registration request, throwing InvalidApplicationMasterRequestException
> "Application Master is already registered".
> I feel like this issue can be resolved by a relatively small update to the
> YARN Java client - e.g. by introducing a new variant of the
> AMRMClientAsync.registerApplicationMaster() that would take the required
> security token (instead of getting it implicitly from the
> UserGroupInformation.getCurrentUser().getCredentials() etc.), or having
> some sort of RM session class that would wrap all data that is currently
> global. I need to think about the elegant API for it.
> What do you guys think? I would love to work on this problem and send you a
> pull request for the upcoming 2.9 release.
> Cheers,
> Sergiy.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message