Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of satish.mittal@inmobi.com
 designates 209.85.213.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGhRFTnTrs49SvDKEqu1jYF8VXLH0N=z3pc1jD-cjZp-_uY66w@mail.gmail.com>
References: 
 <CAFfn_RodLBvPKijjMw_FUFUim-cUksbxjqnEyd9kQdt35QE1CQ@mail.gmail.com>
	<BLU179-W435E8D214B1EF4CD599937A4EE0@phx.gbl>
	<CAGhRFTnTrs49SvDKEqu1jYF8VXLH0N=z3pc1jD-cjZp-_uY66w@mail.gmail.com>
Date: Fri, 8 Aug 2014 12:39:54 +0530
Message-ID: 
 <CAPFf3CqBL6npChX+6csMNgWQRnX4ke_y3aPNaATFfJK23R7hiA@mail.gmail.com>
Subject: Re: [DISCUSS] Recipes in Falcon
From: Satish Mittal <satish.mittal@inmobi.com>
To: dev@falcon.incubator.apache.org
Content-Type: multipart/alternative; boundary=e89a8f838ac5665395050018e41c

--e89a8f838ac5665395050018e41c
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

+1. It is a nice way to roll out pre-canned ETL queries targeting a
particular use-case. This feature will be augmented once pipeline designer
is available.

Recipes can be thought of as a DAG around some basic building blocks (e.g.
replication (HDFS-HDFS/HDFS-Hive/Hive-Hive), retention, data
anonymization/pseudonymization for test data generation etc). Falcon
framework can maintain the common recipes and document their contract. We
could have a recipe repository that is pluggable, wherein user can submit
their recipes with its jars. This also allows user to create high-level
recipes using available ones.


On Fri, Aug 8, 2014 at 12:32 PM, Shwetha GS <shwetha.gs@inmobi.com> wrote:

> +1 for the feature
>
> 1. Is creation of recipe in user space or would that need to be bundled i=
n
> falcon as part of distribution ? How do users provide support artifacts f=
or
> the recipe
> We can have some in-built recipes, but we should also support recipes at
> user space
>
> 2. How do users track execution of instances of a particular recipe. Am
> assuming you can create multiple instances of a single recipe
> We can let the user specify process name
>
>
>
> On Fri, Aug 8, 2014 at 10:55 AM, Srikanth Sundarrajan <sriksun@hotmail.co=
m
> >
> wrote:
>
> > +1 for the capability.
> >
> > Some clarifications.
> >
> > 1. Is creation of recipe in user space or would that need to be bundled
> in
> > falcon as part of distribution ? How do users provide support artifacts
> for
> > the recipe
> > 2. How do users track execution of instances of a particular recipe. Am
> > assuming you can create multiple instances of a single recipe
> > 3. Would simple properties table suffice?
> >
> > Regards
> > Srikanth Sundarrajan
> >
> > > Date: Thu, 7 Aug 2014 19:34:13 -0700
> > > Subject: [DISCUSS] Recipes in Falcon
> > > From: venkatesh@innerzeal.com
> > > To: dev@falcon.incubator.apache.org
> > >
> > > Hi folks,
> > >
> > > Falcon offers many services OOTB and caters to a wide array of use
> cases.
> > > However, there has been many asks that does not fit the functionality
> > > offered by Falcon. I'm proposing that we add recipes to Falcon which =
is
> > > similar to recipes in Whirr and other management solutions such as
> puppet
> > > and chef.
> > >
> > > Overview:
> > > A recipe essentially is a static process template with parameterized
> > > workflow to realize a specific use case. For example:
> > >
> > > * replicating directories from one HDFS cluster to another (not timed
> > > partitions)
> > > * replicating hive metadata (database, table, views, etc.)
> > > * replicating between HDFS and Hive - either way
> > > * anonymization of data based on schema
> > > * data masking
> > > * etc.
> > >
> > > Proposal:
> > > Falcon provides a Process abstraction that encapsulates the
> configuration
> > > for a user workflow with scheduling controls. All recipes can be
> modeled
> > > as a Process with in Falcon which executes the user workflow
> > > periodically. The process and its associated workflow are
> parameterized.
> > > The user will provide a properties file with name value pairs that ar=
e
> > > substituted by falcon before scheduling it.
> > >
> > > The CLI would look something like this:
> > > falcon -recipe $recipe_name -properties $properties_file
> > >
> > > Thoughts?
> > >
> > > --
> > > Regards,
> > > Venkatesh
> > >
> > > =E2=80=9CPerfection (in design) is achieved not when there is nothing=
 more to
> > add,
> > > but rather when there is nothing more to take away.=E2=80=9D
> > > - Antoine de Saint-Exup=C3=A9ry
> >
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for th=
e
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileg=
ed
> information. If you are not the intended recipient you are hereby notifie=
d
> that any disclosure, copying, distribution or taking any action in relian=
ce
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmissi=
on
> of the information contained in this communication nor for any delay in i=
ts
> receipt.
>

--=20
_____________________________________________________________
The information contained in this communication is intended solely for the=
=20
use of the individual or entity to whom it is addressed and others=20
authorized to receive it. It may contain confidential or legally privileged=
=20
information. If you are not the intended recipient you are hereby notified=
=20
that any disclosure, copying, distribution or taking any action in reliance=
=20
on the contents of this information is strictly prohibited and may be=20
unlawful. If you have received this communication in error, please notify=
=20
us immediately by responding to this email and then delete it from your=20
system. The firm is neither liable for the proper and complete transmission=
=20
of the information contained in this communication nor for any delay in its=
=20
receipt.

--e89a8f838ac5665395050018e41c--