incubator-oozie-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Islam <misla...@yahoo.com>
Subject Re: Oozie workflow split in files?
Date Thu, 05 Apr 2012 19:10:26 GMT
It is not possible to put multiple wf xml in one file other than using sub-wf action.

How to specify wf.xml name?
This is the logic.
if(apppath is a file) {
        it considers that as workflow.
}else {
        appptah += "workflow.xml"
}
So, in short,  if you give the app_path like this hdfs://user/joe/my_wf/w1.xml, oozie will
consider that as your workflow otherwise it will append "workflow.xml"

I'm sorry that it is not documented in any place. It is there for a while.
Can someone please create a JIRA on this? Otherwise I will create one later.

Regards,
Mohammad



________________________________
From: Deepika Khera <dkhera@lyris.com>
To: "oozie-users@incubator.apache.org" <oozie-users@incubator.apache.org>; Mohammad
Islam <mislam77@yahoo.com> 
Sent: Thursday, April 5, 2012 11:04 AM
Subject: Re: Oozie workflow split in files?

These jobs are actually a part of a single workflow, though they can be
logically grouped into categories. Since the number of jobs is quite
big, I wanted to split them into separate files for simplicity.

In sub-workflow schema I only see the possibility of elements -
app-path, propagate-configuration and configuration . 
Could you please post a small sample of how these multiple workflow
files could be embedded into a single workflow? 

Oh and as Max pointed out I am not aware of a way to rename workflow.xml
to some other name. Is there a configuration that can be used for this
in oozie-site.xml?

Thanks,
Deepika

On Wed, 2012-04-04 at 23:30 -0700, Mohammad Islam wrote:
> Hi Deepika,
> Yes you can write a WF application with multiple workflow xml files in the same directory
say w1.xml, w2.xml, .., w9.xml=A0 and w_combo.xml.
> In w_combo.xml, you can execute sub-wf actions for each wf?.xmlin any sequence (such
as forking).
> 
> Do you see real value to trigger multiple WFs from one coordinator? The question is in
what sequence you want to launch those WFs? The reason, we don't want to support another DAG
engine like concept at coordinator level.
> 
> Regards,
> Mohammad
> 
> 
> ________________________________
> From: Deepika Khera <dkhera@lyris.com>
> To: "oozie-users@incubator.apache.org" <oozie-users@incubator.apache.org> 
> Sent: Wednesday, April 4, 2012 5:01 PM
> Subject: Re: Oozie workflow split in files?
> 
> Thanks Max. Yeah, seems like a reasonable way to achieve this.
> I can split my cumbersome "hourly" workflow, into separate workflows
> applications and then a main workflow can bundle them up as subworkflows
> and trigger them all at once(through a common coordinator maybe).
> I tried it out (though without the coordinator at this point) and it
> seems to work fine.
> There is an overhead of deploying these as separate workflows rather
> than having them all together in a single workflow application, but that
> should be ok (unless there is another way to do this that we are unaware
> of)
> 
> Thanks!
> Deepika
> 
> 
> 
> On Wed, 2012-04-04 at 16:16 -0700, Maxime Petazzoni wrote:
> > Deepika,
> > 
> > * Deepika Khera <dkhera@lyris.com> [2012-04-04 16:09:43]:
> > 
> > > I have about 70 jobs to run as a part of an hourly workflow. I was
> > > wondering if it will be possible to split these into separate workflow
> > > files (like each sub-workflow in a separate file), so it is not one
> > > large file describing all the workflow jobs.
> > > Is there a way to achieve something like this?
> > > Currently , I can find only examples of defining sub-workflows in the
> > > same file.
> > 
> > Are these jobs independant? You can split the independant blocks into
> > distincts workflows and coordinators, and submit them all at once as a
> > bundle maybe?
> > 
> > Note that I'm not sure this is really the "best practice" for this
> > situation. I'm still trying to figure a lot of things out with Oozie
> > myself :)
> > 
> > /Max

Mime
View raw message