falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Seetharam Venkatesh <venkat...@innerzeal.com>
Subject [DISCUSS] Recipes in Falcon
Date Fri, 08 Aug 2014 02:34:13 GMT
Hi folks,

Falcon offers many services OOTB and caters to a wide array of use cases.
However, there has been many asks that does not fit the functionality
offered by Falcon. I'm proposing that we add recipes to Falcon which is
similar to recipes in Whirr and other management solutions such as puppet
and chef.

Overview:
A recipe essentially is a static process template with parameterized
workflow to realize a specific use case. For example:

* replicating directories from one HDFS cluster to another (not timed
partitions)
* replicating hive metadata (database, table, views, etc.)
* replicating between HDFS and Hive - either way
* anonymization of data based on schema
* data masking
* etc.

Proposal:
Falcon provides a Process abstraction that encapsulates the configuration
for a user workflow with scheduling controls. All recipes can be modeled
as a Process with in Falcon which executes the user workflow
periodically. The process and its associated workflow are parameterized.
The user will provide a properties file with name value pairs that are
substituted by falcon before scheduling it.

The CLI would look something like this:
falcon -recipe $recipe_name -properties $properties_file

Thoughts?

-- 
Regards,
Venkatesh

“Perfection (in design) is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.”
- Antoine de Saint-Exupéry

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message