falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharad Agarwal <sha...@apache.org>
Subject Re: [DISCUSS] Recipes in Falcon
Date Fri, 08 Aug 2014 06:47:10 GMT
big +1 this will be very useful


On Fri, Aug 8, 2014 at 8:04 AM, Seetharam Venkatesh <venkatesh@innerzeal.com
> wrote:

> Hi folks,
>
> Falcon offers many services OOTB and caters to a wide array of use cases.
> However, there has been many asks that does not fit the functionality
> offered by Falcon. I'm proposing that we add recipes to Falcon which is
> similar to recipes in Whirr and other management solutions such as puppet
> and chef.
>
> Overview:
> A recipe essentially is a static process template with parameterized
> workflow to realize a specific use case. For example:
>
> * replicating directories from one HDFS cluster to another (not timed
> partitions)
> * replicating hive metadata (database, table, views, etc.)
> * replicating between HDFS and Hive - either way
> * anonymization of data based on schema
> * data masking
> * etc.
>
> Proposal:
> Falcon provides a Process abstraction that encapsulates the configuration
> for a user workflow with scheduling controls. All recipes can be modeled
> as a Process with in Falcon which executes the user workflow
> periodically. The process and its associated workflow are parameterized.
> The user will provide a properties file with name value pairs that are
> substituted by falcon before scheduling it.
>
> The CLI would look something like this:
> falcon -recipe $recipe_name -properties $properties_file
>
> Thoughts?
>
> --
> Regards,
> Venkatesh
>
> “Perfection (in design) is achieved not when there is nothing more to add,
> but rather when there is nothing more to take away.”
> - Antoine de Saint-Exupéry
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message