falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-634) Add recipes in Falcon
Date Tue, 27 Jan 2015 15:12:35 GMT

    [ https://issues.apache.org/jira/browse/FALCON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293665#comment-14293665
] 

Srikanth Sundarrajan commented on FALCON-634:
---------------------------------------------

[~sowmyaramesh], To control recipe through client makes it very hard for users to discover
what recipes are available for use. To make this client based feature, will greatly limit
the extensibility of the feature. As central service provide the falcon administrator can
make more and more recipes available to users without having to coordinate with them. Besides
opening the possibility of rolling out bug fixes more easily to the recipe logic. Being in
client space, makes all of this very hard.

If users are to make changes to template, while it seems like putting the power in the hands
of the user, the behavior for the recipe may not be consistent causing trouble from a debugging
& administration stand point, besides placing the burden on the users to have greater
knowledge about the recipe and its implementation.


> Add recipes in Falcon
> ---------------------
>
>                 Key: FALCON-634
>                 URL: https://issues.apache.org/jira/browse/FALCON-634
>             Project: Falcon
>          Issue Type: Improvement
>    Affects Versions: 0.6
>            Reporter: Venkatesh Seetharam
>              Labels: recipes
>
> Falcon offers many services OOTB and caters to a wide array of use cases. However, there
has been many asks that does not fit the functionality offered by Falcon. I'm proposing that
we add recipes to Falcon which is similar to recipes in Whirr and other management solutions
such as puppet and chef.
> Overview:
> A recipe essentially is a static process template with parameterized workflow to realize
a specific use case. For example:
> * replicating directories from one HDFS cluster to another (not timed partitions)
> * replicating hive metadata (database, table, views, etc.)
> * replicating between HDFS and Hive - either way
> * anonymization of data based on schema
> * data masking
> * etc.
> Proposal:
> Falcon provides a Process abstraction that encapsulates the configuration 
> for a user workflow with scheduling controls. All recipes can be modeled 
> as a Process with in Falcon which executes the user workflow 
> periodically. The process and its associated workflow are parameterized. The user will
provide a properties file with name value pairs that are substituted by falcon before scheduling
it.
> This is a client side concept. The server does not know about a recipe but only accepts
the cooked recipe as a process entity. 
> The CLI would look something like this:
> falcon -recipe $recipe_name -properties $properties_file
> Recipes will reside inside addons (contrib) with source code and will have an option
to package 'em.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message