falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Seetharam Venkatesh <venkat...@innerzeal.com>
Subject Re: [DISCUSS] Recipes in Falcon
Date Fri, 22 Aug 2014 20:13:12 GMT
Created https://issues.apache.org/jira/browse/FALCON-634


On Tue, Aug 19, 2014 at 10:18 PM, <jb@nanthrax.net> wrote:

> Hi guys,
>
> It sounds good. I'm back after a "long" period on customers engagements ;)
>
> I think server-side for recipe is interesting as it would allow ready to
> use webapp stuff.
>
> Regarding the packaging, it could be part of falcon-extra: in other Apache
> projects (like Camel, ServiceMix, etc), when I have a non Apache compliant
> "extension" (for instance in terms of license), I put it in *-extra.
> It's hosted on github, approved by Apache but not under the Apache "scope"
> (about release process vote, license, etc).
>
> Regards
> JB
>
>
> On 2014-08-20 00:17, Seetharam Venkatesh wrote:
>
>> There are a couple of questions before I create a jira.
>>
>> Is this a client-side only concept?
>> Do we provide a server-side recipe tool as well so web apps can use this
>> instead. The downside to using the client side api is that it expects
>> kinit
>> to have been done.
>>
>> How do we handle packaging?
>> There could be recipe dependent jars, libs and this cannot be released in
>> apache.
>>
>> Thoughts?
>>
>>
>>
>> On Fri, Aug 8, 2014 at 2:46 PM, Seetharam Venkatesh <
>> venkatesh@innerzeal.com
>>
>>> wrote:
>>>
>>
>>
>>>
>>>
>>> On Thu, Aug 7, 2014 at 10:25 PM, Srikanth Sundarrajan <
>>> sriksun@hotmail.com
>>> > wrote:
>>>
>>>  +1 for the capability.
>>>>
>>>> Some clarifications.
>>>>
>>>> 1. Is creation of recipe in user space or would that need to be bundled
>>>> in falcon as part of distribution ?
>>>>
>>>
>>> The thinking is to have it bundled in falcon as part of the distribution
>>> to start with.
>>>
>>> User space is quite nice but falcon recipe can look at the templates dir
>>> to find one and then substitute the properties. This might be possible
>>> too.
>>>
>>>
>>>  How do users provide support artifacts for the recipe
>>>>
>>>>  All artifacts must be on HDFS.
>>>
>>>
>>>  2. How do users track execution of instances of a particular recipe. Am
>>>> assuming you can create multiple instances of a single recipe
>>>>
>>>>  Yes, name can also be templated and parameterized.
>>>
>>>
>>>  3. Would simple properties table suffice?
>>>>
>>>>  Thats the thinking. Sowmya has prototyped DR running on a source
>>> cluster.
>>>
>>>
>>>
>>>> Regards
>>>> Srikanth Sundarrajan
>>>>
>>>> > Date: Thu, 7 Aug 2014 19:34:13 -0700
>>>> > Subject: [DISCUSS] Recipes in Falcon
>>>> > From: venkatesh@innerzeal.com
>>>> > To: dev@falcon.incubator.apache.org
>>>> >
>>>> > Hi folks,
>>>> >
>>>> > Falcon offers many services OOTB and caters to a wide array of use
>>>> cases.
>>>> > However, there has been many asks that does not fit the functionality
>>>> > offered by Falcon. I'm proposing that we add recipes to Falcon which
>>>> is
>>>> > similar to recipes in Whirr and other management solutions such as
>>>> puppet
>>>> > and chef.
>>>> >
>>>> > Overview:
>>>> > A recipe essentially is a static process template with parameterized
>>>> > workflow to realize a specific use case. For example:
>>>> >
>>>> > * replicating directories from one HDFS cluster to another (not timed
>>>> > partitions)
>>>> > * replicating hive metadata (database, table, views, etc.)
>>>> > * replicating between HDFS and Hive - either way
>>>> > * anonymization of data based on schema
>>>> > * data masking
>>>> > * etc.
>>>> >
>>>> > Proposal:
>>>> > Falcon provides a Process abstraction that encapsulates the
>>>> configuration
>>>> > for a user workflow with scheduling controls. All recipes can be
>>>> modeled
>>>> > as a Process with in Falcon which executes the user workflow
>>>> > periodically. The process and its associated workflow are
>>>> parameterized.
>>>> > The user will provide a properties file with name value pairs that are
>>>> > substituted by falcon before scheduling it.
>>>> >
>>>> > The CLI would look something like this:
>>>> > falcon -recipe $recipe_name -properties $properties_file
>>>> >
>>>> > Thoughts?
>>>> >
>>>> > --
>>>> > Regards,
>>>> > Venkatesh
>>>> >
>>>> > “Perfection (in design) is achieved not when there is nothing more
to
>>>> add,
>>>> > but rather when there is nothing more to take away.”
>>>> > - Antoine de Saint-Exupéry
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Venkatesh
>>>
>>> “Perfection (in design) is achieved not when there is nothing more to
>>> add,
>>> but rather when there is nothing more to take away.”
>>> - Antoine de Saint-Exupéry
>>>
>>>


-- 
Regards,
Venkatesh

“Perfection (in design) is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.”
- Antoine de Saint-Exupéry

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message