falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-2182) Add support for Falcon user extensions
Date Wed, 16 Nov 2016 07:48:58 GMT

    [ https://issues.apache.org/jira/browse/FALCON-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669745#comment-15669745

Pallavi Rao commented on FALCON-2182:

1. Noted that the User Parameter Template returns a collection of properties. Is this fixed
to be a collection of properties ? 
It is just an illustrative example. We are not limiting/enforcing any format/structure for
user parameters. Thought we should leave it up to the extension developer to decide.

4.  Are we thinking of extensions as a convenient way for generating entities or are we thinking
of extensions as a atomic unit of reusable functionality?
Answering this first as it easier to answer other questions based on our choice here. We are
looking at extensions as a convenient way for generating entities. We are not looking at it
as an atomic unit. The reason is 2-fold: 
1. Even if we treat the entities as an atomic unit and allow operations on it as a "unit",
these are eventually individual entities. We will need to have mechanisms to stop the user
from operating/modifying them individually. A lot more complexity will come into picture if
we were to restrict a user from operating on individual entities "belonging" to an extension.
So, plan to keep it simple for now.
2. Ensuring/Guaranteeing atomicity on entity operations such as update, delete, schedule,
suspend/resume across all entities may be difficult too. How do we "rollback", if lets say
update fails for all, but, one entity. User will have to fall-back to performing that operation
on individual entity, in case of failures.

2. Why is preparation & submission separated out ? 
Did give this some thought on do we want to lock down on the generated entities or allow users
to modify them. The "prepare only" option was for advanced users who know what they are doing.
For example, retry option for a process might be part of the generated definition. But, a
user might want to change the periodicity of the same. If we were to take these also as parameters,
there will be an explosion of parameters. The parameters might turn out to be as verbose as
the definitions themselves. Hence chose to provide the "prepare" only option in lieu of keeping
the user parameters from exploding.

Understand that this might allow users to tamper with the definitions. But, we can assume
that the tampering is intentional and user understands the repercussions of it.

3. Can extensions be removed while being referred to ?
Disabling an extension sounds like a good option. If user "unregisters", we validate if no
instances of that extension exist. Disabling/deprecating an extension, will ensure no further
instances are created. Will make the change.

> Add support for Falcon user extensions
> --------------------------------------
>                 Key: FALCON-2182
>                 URL: https://issues.apache.org/jira/browse/FALCON-2182
>             Project: Falcon
>          Issue Type: New Feature
>          Components: extensions
>            Reporter: Pallavi Rao
>         Attachments: FalconUserExtensions.pdf
> Currently server-side trusted extensions are supported (see FALCON-634 for more details).
> Need to extend the Falcon Extensions to support extensions built by users. For example,
lets say a user wants to add an extension for "data-quality check". This shouldn't require
it to be compiled and tested with Falcon build. User should be able to add it on-the-fly (while
the Falcon Server is running) and other users should be able to discover it, and "instantiate"
this new extension.

This message was sent by Atlassian JIRA

View raw message