hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bo Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4495) Workflow Application Master in YARN
Date Mon, 30 Jul 2012 20:59:33 GMT
Bo Wang created MAPREDUCE-4495:
----------------------------------

             Summary: Workflow Application Master in YARN
                 Key: MAPREDUCE-4495
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495
             Project: Hadoop Map/Reduce
          Issue Type: New Feature
    Affects Versions: 2.0.0-alpha
            Reporter: Bo Wang
            Assignee: Bo Wang


It is useful to have a workflow application master, which will be capable of running a DAG
of jobs. The workflow client submits a DAG request to the AM and then the AM will manage the
life cycle of this application in terms of requesting the needed resources from the RM, and
starting, monitoring and retrying the application's individual tasks.

Compared to running Oozie with the current MapReduce Application Master, these are some of
the advantages:
 - Less number of consumed resources, since only one application master will be spawned for
the whole workflow.
 - Reuse of resources, since the same resources can be used by multiple consecutive jobs in
the workflow (no need to request/wait for resources for every individual job from the central
RM).
 - More optimization opportunities in terms of collective resource requests.
 - Optimization opportunities in terms of rewriting and composing jobs in the workflow (e.g.
pushing down Mappers).
 - This Application Master can be reused/extended by higher systems like Pig and hive to provide
an optimized way of running their workflows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message