falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-965) Open up life cycle stage implementation within Falcon for extension
Date Tue, 22 Sep 2015 11:54:04 GMT

    [ https://issues.apache.org/jira/browse/FALCON-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902445#comment-14902445

Pallavi Rao commented on FALCON-965:

[~ajayyadav], I have put some comments on RB too. However, I had couple of questions that
I thought warranted discussion on the JIRA. So, here they are :

1. Lifecycle is not really a Workflow Engine. Rather, a workflow engine should understand
Lifecycle like it understands Entity (Feed and Process) now. If Lifecycle is a separate engine,
has the following issues:
a. This is meant to be extensible by user. We don't want users to be building co-ords, bundles.
It can be more like, the builder returns (or points) to the user-workflow to execute (and
properties) for a given policy.
b. Plug in other workflow engines (such as our native scheduler)

2. It is not clear to me, what happens when a user specifies both the old retention element
and the new lifecycle one. I'm assuming lifecycle takes precedence. 
a. Will the retention element be used as fallback, in case, lifecycle fails validation?
b. Since only retention is going in now, replication will be specified in the old way. In
this case, the OozieWorkflowEngine will process the feed and generate appropriate bundles.
If retention is specified too, how does it know to ignore it?

> Open up life cycle stage implementation within Falcon for extension
> -------------------------------------------------------------------
>                 Key: FALCON-965
>                 URL: https://issues.apache.org/jira/browse/FALCON-965
>             Project: Falcon
>          Issue Type: New Feature
>    Affects Versions: 0.7
>            Reporter: Srikanth Sundarrajan
>            Assignee: Ajay Yadava
>              Labels: recipes
>             Fix For: 0.8
>         Attachments: FALCON-965-v1.patch, FALCON-965.patch, FalconLifecycle-Designdoc.pdf,
> As it stands Falcon supports replication, generation and eviction lifecycle stages and
plans to support more. This however assumes a certain way of implementing a life cycle function
and changes to these implementation aren't easy, as they are not open for easy extension.
This proposed feature is open this up in Falcon.
> Here is a proposal on how things can possibly be:
> * List of life cycles that Falcon supports would be well known and not extensible
> * Dependency between life cycles are coded up in the falcon server and not necessarily
extensible. (In short adding a new life cycle still requires changes in Falcon)
> * Each Lifecycle in falcon advertises an implementation interface and minimum configuration
interface (for ex. Eviction should expose a way to retrieve the configured time limit for
which data will be available for other life cycle stages to validate. There is no point in
having a process consume last 24 instances of a feed, when the retention will retain only
4 instances)
> * Similar to FALCON-634, life cycle implementation can be dropped in as long as the implementation
interface and configuraion interfaces are adhered to.

This message was sent by Atlassian JIRA

View raw message