Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of psychidris@gmail.com designates
 209.85.215.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BLU179-W931569420E9A4C4B4DDF57A4690@phx.gbl>
References: <BLU179-W931569420E9A4C4B4DDF57A4690@phx.gbl>
From: Idris Ali <psychidris@gmail.com>
Date: Mon, 22 Dec 2014 01:52:04 +0530
Message-ID: 
 <CAHAmRcubFDqVAZzhi_1q3_7RVaaRhP=AgpkRexPE=ndCnLu2BA@mail.gmail.com>
Subject: Re: [DISCUSS] Orchestration in Falcon
To: dev@falcon.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1133d3765dff0f050abfb4ef

--001a1133d3765dff0f050abfb4ef
Content-Type: text/plain; charset=UTF-8

Hi,

I agree that oozie has certain limitation with respect to orchestration,
recently oozie users have raised similar concerns regarding Point 2. (which
is taken care by falcon by extending Oozie EL not for scheduling but at
least for consuming the input set)

Is it not Point 1, 2, 6 and 7 are already solved by using optional input
mechanism in Falcon? I understand that still users need to specify
frequency for the process. A few usecases/examples would really help.

Thanks,
-Idris


On Sun, Dec 21, 2014 at 7:43 PM, Srikanth Sundarrajan <sriksun@hotmail.com>
wrote:

> Hello Team,
>
> Since its inception Falcon has used Oozie for process orchestration as
> well as feed life cycle phase executions, while this has worked reasonably
> and allowed to make higher level capabilities available through Falcon, we
> are increasing seeing scenarios where this is proving to be a limiting
> factor. In its current form, Falcon relies on Oozie for both scheduling and
> for workflow execution, due to which the scheduling is limited to time
> based/cron based scheduling with additional gating conditions on data
> availability. Also this imposes restrictions on datesets being
> periodic/cyclic in nature.
>
> From an orchestration stand point, it would help if we can support
> standard gating / scheduling primitives via Falcon:
>
> 1. Simple periodic scheduling with no gating conditions
> 2. Cron based scheduling (day of week, day of the month, specific hours
> and non-periodic) with no gating conditions
> 3. Availability of new data (assuming monotonically increasing data
> version, availavility of new versions)
> 4. Changes to existing data (reinstatement - similar to late data handling)
> 5. External trigger/notifications
> 6. Availability of specific instances of data as declared as mandatory
> dependency
> 7. Availability of a minimum subset of instances of data declared as
> mandatory depedency (at least 10 hourly instances of a day with 24
> instances for ex)
> 8. Valid combinations of the above.
>
> In this context, I would like to propose that we move away from Oozie for
> the orchestration requirements and have them implemented natively within
> Falcon. It will no doubt make Falcon server bulkier and heavier in both
> code and deployment, but seems like without it, the orchestration within
> Falcon will be limited by capabilities available within Oozie.
>
> Please do note that this suggestion is restricted to the scheduling and
> not to the workflow execution.
>
> Would like to hear from fellow developers and users on what your thoughts
> are. Please do chime in with your views.
>
> Regards
> Srikanth Sundarrajan
>

--001a1133d3765dff0f050abfb4ef--