hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4495) Workflow Application Master in YARN
Date Wed, 10 Oct 2012 19:57:06 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473513#comment-13473513
] 

Robert Joseph Evans commented on MAPREDUCE-4495:
------------------------------------------------

I really do like the idea of having an AM that can run a workflow.  I think that there is
a huge potential here and I want to see this move forward, but the size and scope of this
change is a lot to take in. There are 11,734 lines in the patch.  I realize that a lot of
this was taken from Oozie itself, but then how are we going to keep the two in sync?  What
happens when Oozie finds a bug?  How are we going to be sure that the bug is pulled into mapred?
 I really would prefer to see a more agile approach to these changes, and hopefully some of
them can correspond to MR, YARN, and HDFS splitting apart after 2.0 has stabilized, so Arun's
fears about Hadoop returning to be a project of projects can be alleviated.

Can we look at moving the parts that can be common between Oozie and the workflow AM into
a separate project? That project I would expect to eventually own the complete Workflow AM,
but in the short term it would just provide a place for this workflow library.  In parallel
with that we can move forward and put in a simple AM that allows for the existing JobControl
API to run in an AM.  This would allow us to validate that the MR AM is thread safe, and keep
it that way.  It would also offer a potentially huge benefit to pig which does use that API
currently.  I would expect most of the initial code for this JobControl workflow AM to be
replaced as it moves to use the common workflow library.  

By doing this in an agile fashion it would also allow us to work out a number of potential
issues I see when moving this from Oozie which uses a DB to store its state to a workflow
AM where that is not possible.  By doing an initial simple JobControl AM we can work out some
of the issues with restarting the AM after it crashes.  What is more by keeping the changes
small, it is much more likely to be something that can be merged into branch 2 so that the
branches do not diverge nearly as much.
                
> Workflow Application Master in YARN
> -----------------------------------
>
>                 Key: MAPREDUCE-4495
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>    Affects Versions: 2.0.0-alpha
>            Reporter: Bo Wang
>            Assignee: Bo Wang
>         Attachments: MAPREDUCE-4495-v1.1.patch, MAPREDUCE-4495-v1.patch, MapReduceWorkflowAM.pdf
>
>
> It is useful to have a workflow application master, which will be capable of running
a DAG of jobs. The workflow client submits a DAG request to the AM and then the AM will manage
the life cycle of this application in terms of requesting the needed resources from the RM,
and starting, monitoring and retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master, these are some
of the advantages:
>  - Less number of consumed resources, since only one application master will be spawned
for the whole workflow.
>  - Reuse of resources, since the same resources can be used by multiple consecutive jobs
in the workflow (no need to request/wait for resources for every individual job from the central
RM).
>  - More optimization opportunities in terms of collective resource requests.
>  - Optimization opportunities in terms of rewriting and composing jobs in the workflow
(e.g. pushing down Mappers).
>  - This Application Master can be reused/extended by higher systems like Pig and hive
to provide an optimized way of running their workflows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message