hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "WritingYarnApps" by Arun C Murthy
Date Tue, 20 Sep 2011 21:55:49 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "WritingYarnApps" page has been changed by Arun C Murthy:
http://wiki.apache.org/hadoop/WritingYarnApps

New page:
We describe, at a high-level, the way to implement new Applications for YARN.

=== Concepts & Flow ===
The general concept is: the 'application submission client' submits an 'application' to the
ResourceManager via ClientRMProtocol#getNewApplicationId/ClientRMProtocol#submitApplication.
During the call to ClientRMProtocol#submitApplication you need to provide sufficient information
to the ResourceManager to 'launch' the application's first container i.e. the ApplicationMaster
- see ApplicationSubmissionContext for details. You need to provide app-specific command line,
unix env (optional) etc. - effectively you need to describe the unix process which needs to
be launched for your ApplicationMaster.

The RM then launches the ApplicationMaster (as specified) which then communicates to the RM
via AMRMProtocol. 

First it needs to register (AMRMProtocol#registerApplicationMaster. Then uses AMRMProtocol#allocate
to request/receive for containers for it's task.

To launch a container for its task the AM uses ContainerManager#startContainer (i.e. talks
to NodeManager which implements this x'face) and passes a ContainerLaunchContext which, similar
to ApplicationSubmissionContext, has command-line, env etc.

=== Interfaces ===
The interfaces you'd most like be concerned with are:
ContainerManager -> Protocol for AM to talk to NodeManager to start/stop containers
ClientRMProtocol -> Protocol for job-client (launching program from gateway) to talk to
RM to launch a new application i.e. application master itself
AMRMProtocol -> Protocol for AM to use to register/stop itself and get resources from the
Scheduler

Mime
View raw message