hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Oozie vs. YARN "application"
Date Fri, 18 May 2012 00:49:47 GMT
Currently YARN doesn't offer anything to manage a DAG of applications.

It's fairly easy to implement a DAGApplicationMaster to manage a set of applications (whether
MR or others).


PS: Please use mapreduce-dev@ for technical discussions, general@ is used for project discussions/announcements.

On May 17, 2012, at 5:41 PM, Keith Wiley wrote:

> Just to be clear yarn does offer something described as dag management, but perhaps you
mean it doesn't offer a dag *of* applications, only of mr jobs *within* an application. Is
that what you mean?
> ________________________________________
> Sent from my phone, please excuse my brevity.
> Keith Wiley, kwiley@keithwiley.com, http://keithwiley.com
> Arun C Murthy <acm@hortonworks.com> wrote:
> YARN doesn't yet have a DAGApplicationMaster which can handle a DAG of jobs.
> Conceivable Oozie could offload the DAG management once it's available.
> OTOH, Oozie provides much more than just DAG management - it provides time & data-availability
based scheduling of workflows.
> hth,
> Arun
> On May 17, 2012, at 4:47 PM, Keith Wiley wrote:
>> Hadoop 0.23 (MRv2 or YARN) provides the concept of an "application", which is described
as either a classic single MR job *or* a DAG of such jobs...which at a glance appears redundant
with Oozie's primary purpose.
>> How do they differ? Does 0.23 effectively obviate the service that Oozie provides
or is Oozie more powerful than a YARN "application"...or simply more different from an application
than I have conceptualized to the point that they don't necessarily step on one another's
>> _____________________________________________
>> Keith Wiley kwiley@keithwiley.com keithwiley.com music.keithwiley.com
>> "The easy confidence with which I know another man's religion is folly teaches
>> me to suspect that my own is also."
>> -- Mark Twain
>> _____________________________________________
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/

Arun C. Murthy
Hortonworks Inc.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message