falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From j.@nanthrax.net
Subject Re: [DISCUSS] Recipes in Falcon
Date Wed, 20 Aug 2014 05:18:41 GMT
Hi guys,

It sounds good. I'm back after a "long" period on customers engagements 
;)

I think server-side for recipe is interesting as it would allow ready to 
use webapp stuff.

Regarding the packaging, it could be part of falcon-extra: in other 
Apache projects (like Camel, ServiceMix, etc), when I have a non Apache 
compliant "extension" (for instance in terms of license), I put it in 
*-extra.
It's hosted on github, approved by Apache but not under the Apache 
"scope" (about release process vote, license, etc).

Regards
JB

On 2014-08-20 00:17, Seetharam Venkatesh wrote:
> There are a couple of questions before I create a jira.
> 
> Is this a client-side only concept?
> Do we provide a server-side recipe tool as well so web apps can use 
> this
> instead. The downside to using the client side api is that it expects 
> kinit
> to have been done.
> 
> How do we handle packaging?
> There could be recipe dependent jars, libs and this cannot be released 
> in
> apache.
> 
> Thoughts?
> 
> 
> 
> On Fri, Aug 8, 2014 at 2:46 PM, Seetharam Venkatesh 
> <venkatesh@innerzeal.com
>> wrote:
> 
>> 
>> 
>> 
>> On Thu, Aug 7, 2014 at 10:25 PM, Srikanth Sundarrajan 
>> <sriksun@hotmail.com
>> > wrote:
>> 
>>> +1 for the capability.
>>> 
>>> Some clarifications.
>>> 
>>> 1. Is creation of recipe in user space or would that need to be 
>>> bundled
>>> in falcon as part of distribution ?
>> 
>> The thinking is to have it bundled in falcon as part of the 
>> distribution
>> to start with.
>> 
>> User space is quite nice but falcon recipe can look at the templates 
>> dir
>> to find one and then substitute the properties. This might be possible 
>> too.
>> 
>> 
>>> How do users provide support artifacts for the recipe
>>> 
>> All artifacts must be on HDFS.
>> 
>> 
>>> 2. How do users track execution of instances of a particular recipe. 
>>> Am
>>> assuming you can create multiple instances of a single recipe
>>> 
>> Yes, name can also be templated and parameterized.
>> 
>> 
>>> 3. Would simple properties table suffice?
>>> 
>> Thats the thinking. Sowmya has prototyped DR running on a source 
>> cluster.
>> 
>> 
>>> 
>>> Regards
>>> Srikanth Sundarrajan
>>> 
>>> > Date: Thu, 7 Aug 2014 19:34:13 -0700
>>> > Subject: [DISCUSS] Recipes in Falcon
>>> > From: venkatesh@innerzeal.com
>>> > To: dev@falcon.incubator.apache.org
>>> >
>>> > Hi folks,
>>> >
>>> > Falcon offers many services OOTB and caters to a wide array of use
>>> cases.
>>> > However, there has been many asks that does not fit the functionality
>>> > offered by Falcon. I'm proposing that we add recipes to Falcon which is
>>> > similar to recipes in Whirr and other management solutions such as
>>> puppet
>>> > and chef.
>>> >
>>> > Overview:
>>> > A recipe essentially is a static process template with parameterized
>>> > workflow to realize a specific use case. For example:
>>> >
>>> > * replicating directories from one HDFS cluster to another (not timed
>>> > partitions)
>>> > * replicating hive metadata (database, table, views, etc.)
>>> > * replicating between HDFS and Hive - either way
>>> > * anonymization of data based on schema
>>> > * data masking
>>> > * etc.
>>> >
>>> > Proposal:
>>> > Falcon provides a Process abstraction that encapsulates the
>>> configuration
>>> > for a user workflow with scheduling controls. All recipes can be modeled
>>> > as a Process with in Falcon which executes the user workflow
>>> > periodically. The process and its associated workflow are parameterized.
>>> > The user will provide a properties file with name value pairs that are
>>> > substituted by falcon before scheduling it.
>>> >
>>> > The CLI would look something like this:
>>> > falcon -recipe $recipe_name -properties $properties_file
>>> >
>>> > Thoughts?
>>> >
>>> > --
>>> > Regards,
>>> > Venkatesh
>>> >
>>> > “Perfection (in design) is achieved not when there is nothing more to
>>> add,
>>> > but rather when there is nothing more to take away.”
>>> > - Antoine de Saint-Exupéry
>>> 
>>> 
>> 
>> 
>> 
>> --
>> Regards,
>> Venkatesh
>> 
>> “Perfection (in design) is achieved not when there is nothing more to 
>> add,
>> but rather when there is nothing more to take away.”
>> - Antoine de Saint-Exupéry
>> 

Mime
View raw message