reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Weimer (JIRA)" <>
Subject [jira] [Commented] (REEF-1747) Workaround to allow two AMs in one YARN container
Date Thu, 09 Mar 2017 03:01:38 GMT


Markus Weimer commented on REEF-1747:

Why do you need two AMs in a process? 

We'd like to be able to run a REEF job inside a Spark job. The Spark Driver would use the
new REEF-as-a-Library to run an unmanaged AM to do so.

What's a non-hacky solution?

Fixing the singleton issue in YARN comes to mind.

> Workaround to allow two AMs in one YARN container
> -------------------------------------------------
>                 Key: REEF-1747
>                 URL:
>             Project: REEF
>          Issue Type: Bug
>          Components: REEF Runtime YARN, REEF-Runtime-YARN
>            Reporter: Sergiy Matusevych
>            Assignee: Sergiy Matusevych
>              Labels: workaround
>   Original Estimate: 168h
>  Remaining Estimate: 168h
> The current version of the YARN Java client does not allow YARN to register two Application
Masters running in the same process. Technically, YARN Resource Manager does not care which
process each AM runs in. However, the Java client contains a singleton {{UserGroupInformation}}
that contains the user credentials. This data structure is shared by all AMs, and when REEF
application tries to register the second (Unmanaged) AM, the client library presents to YARN
RM the security token of the *first* AM. YARN rejects such registration request, throwing
{{InvalidApplicationMasterRequestException}} _"Application Master is already registered"_
> A proper fix for this issue would be a patch for Hadoop YARN Java client; we also need
a quick workaround in REEF so we can run REEF-on-REEF and Spark+REEF applications using unpatched
Hadoop libraries.

This message was sent by Atlassian JIRA

View raw message