falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth Sundarrajan <srik...@hotmail.com>
Subject RE: [DISCUSS] Recipes in Falcon
Date Fri, 08 Aug 2014 05:25:50 GMT
+1 for the capability.

Some clarifications.

1. Is creation of recipe in user space or would that need to be bundled in falcon as part
of distribution ? How do users provide support artifacts for the recipe
2. How do users track execution of instances of a particular recipe. Am assuming you can create
multiple instances of a single recipe
3. Would simple properties table suffice?

Srikanth Sundarrajan

> Date: Thu, 7 Aug 2014 19:34:13 -0700
> Subject: [DISCUSS] Recipes in Falcon
> From: venkatesh@innerzeal.com
> To: dev@falcon.incubator.apache.org
> Hi folks,
> Falcon offers many services OOTB and caters to a wide array of use cases.
> However, there has been many asks that does not fit the functionality
> offered by Falcon. I'm proposing that we add recipes to Falcon which is
> similar to recipes in Whirr and other management solutions such as puppet
> and chef.
> Overview:
> A recipe essentially is a static process template with parameterized
> workflow to realize a specific use case. For example:
> * replicating directories from one HDFS cluster to another (not timed
> partitions)
> * replicating hive metadata (database, table, views, etc.)
> * replicating between HDFS and Hive - either way
> * anonymization of data based on schema
> * data masking
> * etc.
> Proposal:
> Falcon provides a Process abstraction that encapsulates the configuration
> for a user workflow with scheduling controls. All recipes can be modeled
> as a Process with in Falcon which executes the user workflow
> periodically. The process and its associated workflow are parameterized.
> The user will provide a properties file with name value pairs that are
> substituted by falcon before scheduling it.
> The CLI would look something like this:
> falcon -recipe $recipe_name -properties $properties_file
> Thoughts?
> -- 
> Regards,
> Venkatesh
> “Perfection (in design) is achieved not when there is nothing more to add,
> but rather when there is nothing more to take away.”
> - Antoine de Saint-Exupéry
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message