reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Byung-Gon Chun (JIRA)" <>
Subject [jira] [Commented] (REEF-1747) Workaround to allow two AMs in one YARN container
Date Thu, 09 Mar 2017 02:03:38 GMT


Byung-Gon Chun commented on REEF-1747:

This problem sounds interesting. 
Why do you need two AMs in a process? 
What's a non-hacky solution?

> Workaround to allow two AMs in one YARN container
> -------------------------------------------------
>                 Key: REEF-1747
>                 URL:
>             Project: REEF
>          Issue Type: Bug
>          Components: REEF Runtime YARN, REEF-Runtime-YARN
>            Reporter: Sergiy Matusevych
>            Assignee: Sergiy Matusevych
>              Labels: workaround
>   Original Estimate: 168h
>  Remaining Estimate: 168h
> The current version of the YARN Java client does not allow YARN to register two Application
Masters running in the same process. Technically, YARN Resource Manager does not care which
process each AM runs in. However, the Java client contains a singleton {{UserGroupInformation}}
that contains the user credentials. This data structure is shared by all AMs, and when REEF
application tries to register the second (Unmanaged) AM, the client library presents to YARN
RM the security token of the *first* AM. YARN rejects such registration request, throwing
{{InvalidApplicationMasterRequestException}} _"Application Master is already registered"_
> A proper fix for this issue would be a patch for Hadoop YARN Java client; we also need
a quick workaround in REEF so we can run REEF-on-REEF and Spark+REEF applications using unpatched
Hadoop libraries.

This message was sent by Atlassian JIRA

View raw message