Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of sriksun@hotmail.com
 designates 65.55.116.85 as permitted sender)
Message-ID: <BLU179-W29B826B46945FDB36047E6A4570@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_76346cd7-b601-43c5-8d72-390c35b5a138_"
From: Srikanth Sundarrajan <sriksun@hotmail.com>
To: "dev@falcon.incubator.apache.org" <dev@falcon.incubator.apache.org>
Subject: RE: [DISCUSS] Orchestration in Falcon
Date: Tue, 23 Dec 2014 20:58:27 +0530
Importance: Normal
In-Reply-To: <dcea3e87f460628cdda77703affa6ab2@nanthrax.net>
References: 
 <BLU179-W931569420E9A4C4B4DDF57A4690@phx.gbl>,<CAFfn_RofEhCQQ8WE2JMHZNz1+zk1oxdTAqJcP8nfb2bxY2RsQw@mail.gmail.com>,<BLU406-EAS56545341CBE37876478FD3A4570@phx.gbl>,<BLU179-W80C5268BDDCDB853C30B8BA4570@phx.gbl>,<CAAbv78eu+bXqC0st-r6j_w9KkkjZpBZcfrvKCdn=8cZrdhOqxA@mail.gmail.com>,<dcea3e87f460628cdda77703affa6ab2@nanthrax.net>
MIME-Version: 1.0

--_76346cd7-b601-43c5-8d72-390c35b5a138_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

@jb=2C There is no doubt merit in mapping them to oozie if possible and if =
extensions are simple and straight forward enough.=20

Also had a quick chat offline with Shwetha and she mentioned about some wor=
k happening in Oozie in this regard. On further digging up=2C found https:/=
/issues.apache.org/jira/browse/OOZIE-1976. This is possibly what Shwetha wa=
s referring to. From the looks of it=2C this tries to address item #7 in th=
e original thread.  May be there are more jiras where additional work such =
as a-periodic datasets is being worked on. Perhaps @Shwetha can throw some =
light on what is being considered and/or how these gating/orchestration use=
 cases can be managed.

Regards
Srikanth Sundarrajan

> Date: Tue=2C 23 Dec 2014 11:06:24 +0100
> From: jb@nanthrax.net
> To: dev@falcon.incubator.apache.org
> Subject: Re: [DISCUSS] Orchestration in Falcon
>=20
> Hi all=2C
>=20
> I second Shwetha there. I think we can achieve such features in Oozie=20
> (with some adaptations).
>=20
> Regards
> JB
>=20
> Le 2014-12-23 10:53=2C Shwetha G S a =E9crit :
> > If we can get rid of oozie entirely=2C yes we can explore other
> > possibilities. But if we are still going to use oozie for DAG=20
> > execution=2C we
> > are going to add add another bottleneck in the whole=20
> > execution(currently=2C
> > falcon is not in the workflow execution path) and I don't think its=20
> > worth
> > it.
> >=20
> > The features that are outlined above are all available in basic forms=20
> > in
> > oozie and it should be easy to enhance them/make them as extension=20
> > points.
> >=20
> >=20
> >=20
> > -Shwetha
> >=20
> > On Tue=2C Dec 23=2C 2014 at 8:12 AM=2C Srikanth Sundarrajan=20
> > <sriksun@hotmail.com>
> > wrote:
> >=20
> >> Here are few more gaps that we ought to solve for while we are on the
> >> subject:
> >>=20
> >> 1. Ability to attach to start & finish events of workflow execution.
> >> Currently we have post processing hook to listen to finish events=2C b=
ut=20
> >> we
> >> do run into scenarios where there are occasional failures with
> >> post-processing and there is potential phase lag in learning about the
> >> events.
> >> 2. Strict enforcement of concurrency control possibly spanning process
> >> boundaries.
> >> 3. Ability to tune how backlogs have to be caught up (old instances to=
=20
> >> be
> >> given higher priority=2C newer instances to be given higher priority=
=2C or=20
> >> some
> >> sort of weights to allow both to make progress at varying rates).=20
> >> There
> >> have been asks for routing current vs older instances to different=20
> >> queues
> >> by users as an alternative.
> >> 4. Ability to have a notion of non-time based feed instances and=20
> >> related
> >> coordination.
> >> 5. Currently keeping track of and managing SLAs is also a challenge=2C=
=20
> >> but
> >> with #1 addressed=2C this might be a lesser concern.
> >>=20
> >> Regards
> >> Srikanth Sundarrajan
> >>=20
> >> > Subject: Re: [DISCUSS] Orchestration in Falcon
> >> > From: sriksun@hotmail.com
> >> > Date: Tue=2C 23 Dec 2014 06:30:30 +0530
> >> > To: dev@falcon.incubator.apache.org
> >> >
> >> > @venkatesh=2C the question really is how do we enable these gating p=
re
> >> conditions. Seems hard enough to add them to oozie=2C but am not=20
> >> intimately
> >> familiar with oozie to comment on how hard or easy it is. Like I=20
> >> responded
> >> to @ajay on the same thread=2C if we are to do away with coordination=
=20
> >> through
> >> oozie=2C we can follow up this discussion with approaches and design.=
=20
> >> Though
> >> I had quartz in my mind=2C wanted to leave that out of discussion to s=
ee=20
> >> if
> >> there is consensus for moving away from oozie coords and implementing=
=20
> >> them
> >> through other means.
> >> >
> >> > Sent from my iPhone
> >> >
> >> > > On 23-Dec-2014=2C at 1:16 am=2C "Seetharam Venkatesh" <
> >> venkatesh@innerzeal.com> wrote:
> >> > >
> >> > > What is the purpose of this decoupling? Why build this into Falcon=
?
> >> > > Scheduling is so common that there are dime a dozen schedulers tod=
ay
> >> and
> >> > > they are all extensible with custom triggers. Making it part of Fa=
lcon
> >> will
> >> > > suffer the same issues that Oozie has today.
> >> > >
> >> > > I'm sorry but I'm a HUGE -1 to this being built into Falcon codeba=
se.
> >> > >
> >> > > However=2C I'm +1 to reusing Quartz scheduler that already exists =
-
> >> stand it
> >> > > up outside or embed it like we do for active MQ.
> >> > >
> >> > > Phase 2 - I'd like to see we write a simple DAG execution layer in
> >> YARN as
> >> > > an app master with out DB and keeps state on HDFS as an alternate =
to
> >> Oozie.
> >> > >
> >> > > Then we will have a nimble falcon which can kick ass.
> >> > >
> >> > >
> >> > > On Sun=2C Dec 21=2C 2014 at 6:13 AM=2C Srikanth Sundarrajan <
> >> sriksun@hotmail.com>
> >> > > wrote:
> >> > >
> >> > >> Hello Team=2C
> >> > >>
> >> > >> Since its inception Falcon has used Oozie for process orchestrati=
on as
> >> > >> well as feed life cycle phase executions=2C while this has worked
> >> reasonably
> >> > >> and allowed to make higher level capabilities available through
> >> Falcon=2C we
> >> > >> are increasing seeing scenarios where this is proving to be a lim=
iting
> >> > >> factor. In its current form=2C Falcon relies on Oozie for both
> >> scheduling and
> >> > >> for workflow execution=2C due to which the scheduling is limited =
to time
> >> > >> based/cron based scheduling with additional gating conditions on =
data
> >> > >> availability. Also this imposes restrictions on datesets being
> >> > >> periodic/cyclic in nature.
> >> > >>
> >> > >> From an orchestration stand point=2C it would help if we can supp=
ort
> >> > >> standard gating / scheduling primitives via Falcon:
> >> > >>
> >> > >> 1. Simple periodic scheduling with no gating conditions
> >> > >> 2. Cron based scheduling (day of week=2C day of the month=2C spec=
ific
> >> hours
> >> > >> and non-periodic) with no gating conditions
> >> > >> 3. Availability of new data (assuming monotonically increasing da=
ta
> >> > >> version=2C availavility of new versions)
> >> > >> 4. Changes to existing data (reinstatement - similar to late data
> >> handling)
> >> > >> 5. External trigger/notifications
> >> > >> 6. Availability of specific instances of data as declared as mand=
atory
> >> > >> dependency
> >> > >> 7. Availability of a minimum subset of instances of data declared=
 as
> >> > >> mandatory depedency (at least 10 hourly instances of a day with 2=
4
> >> > >> instances for ex)
> >> > >> 8. Valid combinations of the above.
> >> > >>
> >> > >> In this context=2C I would like to propose that we move away from=
 Oozie
> >> for
> >> > >> the orchestration requirements and have them implemented natively
> >> within
> >> > >> Falcon. It will no doubt make Falcon server bulkier and heavier i=
n
> >> both
> >> > >> code and deployment=2C but seems like without it=2C the orchestra=
tion
> >> within
> >> > >> Falcon will be limited by capabilities available within Oozie.
> >> > >>
> >> > >> Please do note that this suggestion is restricted to the scheduli=
ng
> >> and
> >> > >> not to the workflow execution.
> >> > >>
> >> > >> Would like to hear from fellow developers and users on what your
> >> thoughts
> >> > >> are. Please do chime in with your views.
> >> > >>
> >> > >> Regards
> >> > >> Srikanth Sundarrajan
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Regards=2C
> >> > > Venkatesh
> >> > >
> >> > > =93Perfection (in design) is achieved not when there is nothing mo=
re to
> >> add=2C
> >> > > but rather when there is nothing more to take away.=94
> >> > > - Antoine de Saint-Exup=E9ry
> >>=20
> >>=20
 		 	   		  =

--_76346cd7-b601-43c5-8d72-390c35b5a138_--