falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesh Seetharam (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-634) Add recipes in Falcon
Date Fri, 12 Sep 2014 22:28:34 GMT

    [ https://issues.apache.org/jira/browse/FALCON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132223#comment-14132223

Venkatesh Seetharam commented on FALCON-634:

I think some sound very good ideas but others look like we are moving away from client to
server side. We had decided to keep recipes a client side concept.

bq. we should allow recipe to have a RecipeBuilder interface with a build method

bq. List of recipes can be maintained in the server in a shared location (preferably HDFS)

Makes sense

bq.  the prism server can be configured to point to the correct recipe repository folder
I'm not sure why? Why cant the client do this? What if you running in embedded mode?

bq. Author of a recipe can build the recipe archive and drop this in the location
This can be a client side too instead of prism.

bq. Recipe Listing, Description, 
Awesome, very useful, why did I not think of this. :-)

bq. Variable arguments as accepted by the RecipeBuilder
What we have now would suffice and users could use properties file no? This could turn out
bad and unmanageable as we have seen with Sqoop-1.

But, this is a good direction and lets do more thinking around these for the second version.

> Add recipes in Falcon
> ---------------------
>                 Key: FALCON-634
>                 URL: https://issues.apache.org/jira/browse/FALCON-634
>             Project: Falcon
>          Issue Type: Improvement
>    Affects Versions: 0.6
>            Reporter: Venkatesh Seetharam
>              Labels: recipes
> Falcon offers many services OOTB and caters to a wide array of use cases. However, there
has been many asks that does not fit the functionality offered by Falcon. I'm proposing that
we add recipes to Falcon which is similar to recipes in Whirr and other management solutions
such as puppet and chef.
> Overview:
> A recipe essentially is a static process template with parameterized workflow to realize
a specific use case. For example:
> * replicating directories from one HDFS cluster to another (not timed partitions)
> * replicating hive metadata (database, table, views, etc.)
> * replicating between HDFS and Hive - either way
> * anonymization of data based on schema
> * data masking
> * etc.
> Proposal:
> Falcon provides a Process abstraction that encapsulates the configuration 
> for a user workflow with scheduling controls. All recipes can be modeled 
> as a Process with in Falcon which executes the user workflow 
> periodically. The process and its associated workflow are parameterized. The user will
provide a properties file with name value pairs that are substituted by falcon before scheduling
> This is a client side concept. The server does not know about a recipe but only accepts
the cooked recipe as a process entity. 
> The CLI would look something like this:
> falcon -recipe $recipe_name -properties $properties_file
> Recipes will reside inside addons (contrib) with source code and will have an option
to package 'em.

This message was sent by Atlassian JIRA

View raw message