falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesh Seetharam (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-369) Refactor workflow builder
Date Fri, 21 Mar 2014 23:06:44 GMT

    [ https://issues.apache.org/jira/browse/FALCON-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13943701#comment-13943701

Venkatesh Seetharam commented on FALCON-369:

This is a very useful thing to do and thanks for taking the initiative [~shwethags]. I haven't
taken a deeper look at the patch but I have a few thoughts and some may be a stretch:

* The current implementation is oozie centric and this also looks like that at the outset

* Would it be better to approach this from Falcon entities, lifecycle policies, etc. and then
map that to an interface, one of which will be a oozie implementation
SchedulableEntity is a first class interface - There is a corresponding Builder for Oozie
LifecyclePolicy - Specific implementations for Replication, retention, etc. which will have
Builders that will materialize workflow schedules and wf actions
Let me think more and propose a set of interfaces.
* Have another implementation for azkaban or Spring batch so we get the APIs right. I can
volunteer to write this
* Can we use the name OrchestrationWorkflowBuilder - typically users get confused with orchestration
workflow vs user workflow

May be creating subtasks and adding this incrementally might help to drive this collaboratively.

Thanks again.

> Refactor workflow builder
> -------------------------
>                 Key: FALCON-369
>                 URL: https://issues.apache.org/jira/browse/FALCON-369
>             Project: Falcon
>          Issue Type: Improvement
>            Reporter: Shwetha G S
>            Assignee: Shwetha G S
>         Attachments: FALCON-369.patch
> Currently, feed/process workflow builder is a single class which handles all different
cases of lifecycles, storage types, workflow engines and building all oozie entities(workflow,
coord and bundle). This is not readable and difficult to maintain. This needs some re-factoring.
> Approach:
> Maintain different builders for
> 1. oozie entities - workflow, coord and bundle. 
> 2. entity types - feed and process
> 3. lifecycle - process, retention and replication
> 4. workflow engines - oozie, pig and hive

This message was sent by Atlassian JIRA

View raw message