incubator-oozie-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepika Khera <dkh...@lyris.com>
Subject Re: Oozie workflow split in files?
Date Thu, 05 Apr 2012 21:57:10 GMT
Thanks Mohammad. I certainly planned on using the sub-wf action, just
dint know how to specify it as a workflow file Vs the app path. What you
explained will be perfect for what I need to do.

Deepika


On Thu, 2012-04-05 at 12:10 -0700, Mohammad Islam wrote:
> It is not possible to put multiple wf xml in one file other than using sub-wf action.
> 
> How to specify wf.xml name?
> This is the logic.
> if(apppath is a file) {
>         it considers that as workflow.
> }else {
>         appptah += "workflow.xml"
> }
> So, in short,  if you give the app_path like this hdfs://user/joe/my_wf/w1.xml, oozie
will consider that as your workflow otherwise it will append "workflow.xml"
> 
> I'm sorry that it is not documented in any place. It is there for a while.
> Can someone please create a JIRA on this? Otherwise I will create one later.
> 
> Regards,
> Mohammad
> 
> 
> 
> ________________________________
> From: Deepika Khera <dkhera@lyris.com>
> To: "oozie-users@incubator.apache.org" <oozie-users@incubator.apache.org>; Mohammad
Islam <mislam77@yahoo.com> 
> Sent: Thursday, April 5, 2012 11:04 AM
> Subject: Re: Oozie workflow split in files?
> 
> These jobs are actually a part of a single workflow, though they can be
> logically grouped into categories. Since the number of jobs is quite
> big, I wanted to split them into separate files for simplicity.
> 
> In sub-workflow schema I only see the possibility of elements -
> app-path, propagate-configuration and configuration . 
> Could you please post a small sample of how these multiple workflow
> files could be embedded into a single workflow? 
> 
> Oh and as Max pointed out I am not aware of a way to rename workflow.xml
> to some other name. Is there a configuration that can be used for this
> in oozie-site.xml?
> 
> Thanks,
> Deepika
> 
> On Wed, 2012-04-04 at 23:30 -0700, Mohammad Islam wrote:
> > Hi Deepika,
> > Yes you can write a WF application with multiple workflow xml files in the same
directory say w1.xml, w2.xml, .., w9.xml=A0 and w_combo.xml.
> > In w_combo.xml, you can execute sub-wf actions for each wf?.xmlin any sequence (such
as forking).
> > 
> > Do you see real value to trigger multiple WFs from one coordinator? The question
is in what sequence you want to launch those WFs? The reason, we don't want to support another
DAG engine like concept at coordinator level.
> > 
> > Regards,
> > Mohammad
> > 
> > 
> > ________________________________
> > From: Deepika Khera <dkhera@lyris.com>
> > To: "oozie-users@incubator.apache.org" <oozie-users@incubator.apache.org>

> > Sent: Wednesday, April 4, 2012 5:01 PM
> > Subject: Re: Oozie workflow split in files?
> > 
> > Thanks Max. Yeah, seems like a reasonable way to achieve this.
> > I can split my cumbersome "hourly" workflow, into separate workflows
> > applications and then a main workflow can bundle them up as subworkflows
> > and trigger them all at once(through a common coordinator maybe).
> > I tried it out (though without the coordinator at this point) and it
> > seems to work fine.
> > There is an overhead of deploying these as separate workflows rather
> > than having them all together in a single workflow application, but that
> > should be ok (unless there is another way to do this that we are unaware
> > of)
> > 
> > Thanks!
> > Deepika
> > 
> > 
> > 
> > On Wed, 2012-04-04 at 16:16 -0700, Maxime Petazzoni wrote:
> > > Deepika,
> > > 
> > > * Deepika Khera <dkhera@lyris.com> [2012-04-04 16:09:43]:
> > > 
> > > > I have about 70 jobs to run as a part of an hourly workflow. I was
> > > > wondering if it will be possible to split these into separate workflow
> > > > files (like each sub-workflow in a separate file), so it is not one
> > > > large file describing all the workflow jobs.
> > > > Is there a way to achieve something like this?
> > > > Currently , I can find only examples of defining sub-workflows in the
> > > > same file.
> > > 
> > > Are these jobs independant? You can split the independant blocks into
> > > distincts workflows and coordinators, and submit them all at once as a
> > > bundle maybe?
> > > 
> > > Note that I'm not sure this is really the "best practice" for this
> > > situation. I'm still trying to figure a lot of things out with Oozie
> > > myself :)
> > > 
> > > /Max



Mime
View raw message