falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkat Ranganathan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-634) Add recipes in Falcon
Date Thu, 28 Jan 2016 03:37:39 GMT

    [ https://issues.apache.org/jira/browse/FALCON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120698#comment-15120698
] 

Venkat Ranganathan commented on FALCON-634:
-------------------------------------------

Myself, [~sowmyaramesh] and [~yzheng-hortonworks]  discussed this today.

Basically
*  There should be two classes of recipes - core recipes - are those that are designed, developed
and released as part of Falcon - these are essentially upto-date with the release of Falcon,
have regression and integration tests that are run as part of the falcon release and thoroughly
tested.    These qualify to run within the Falcon server and typically live under the _addons_
directory.   Examples are the HDFS and Hive mirroring, and future such addons

* Optional recipes are those that are customer/administrator written, not shipped with the
product and may need updates with newer versions of the product.   Their execution model 
(whether they get scheduled in a managed workflow or on a separate process) are things that
need further discussion and clarity.

The packaging of the recipes for execution, their directory structure, how they can be introspected
so that a UI can render the invocation of these REST APIs are common to both core and optional
recipes.

There will be a new design document that we will  publish.   Just providing the gist of the
discussion to kick off early discussions.

> Add recipes in Falcon
> ---------------------
>
>                 Key: FALCON-634
>                 URL: https://issues.apache.org/jira/browse/FALCON-634
>             Project: Falcon
>          Issue Type: New Feature
>    Affects Versions: 0.6
>            Reporter: Venkatesh Seetharam
>              Labels: recipes
>
> Falcon offers many services OOTB and caters to a wide array of use cases. However, there
has been many asks that does not fit the functionality offered by Falcon. I'm proposing that
we add recipes to Falcon which is similar to recipes in Whirr and other management solutions
such as puppet and chef.
> Overview:
> A recipe essentially is a static process template with parameterized workflow to realize
a specific use case. For example:
> * replicating directories from one HDFS cluster to another (not timed partitions)
> * replicating hive metadata (database, table, views, etc.)
> * replicating between HDFS and Hive - either way
> * anonymization of data based on schema
> * data masking
> * etc.
> Proposal:
> Falcon provides a Process abstraction that encapsulates the configuration 
> for a user workflow with scheduling controls. All recipes can be modeled 
> as a Process with in Falcon which executes the user workflow 
> periodically. The process and its associated workflow are parameterized. The user will
provide a properties file with name value pairs that are substituted by falcon before scheduling
it.
> This is a client side concept. The server does not know about a recipe but only accepts
the cooked recipe as a process entity. 
> The CLI would look something like this:
> falcon -recipe $recipe_name -properties $properties_file
> Recipes will reside inside addons (contrib) with source code and will have an option
to package 'em.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message