falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satish Mittal <satish.mit...@inmobi.com>
Subject Re: [DISCUSS] Recipes in Falcon
Date Fri, 08 Aug 2014 07:09:54 GMT
+1. It is a nice way to roll out pre-canned ETL queries targeting a
particular use-case. This feature will be augmented once pipeline designer
is available.

Recipes can be thought of as a DAG around some basic building blocks (e.g.
replication (HDFS-HDFS/HDFS-Hive/Hive-Hive), retention, data
anonymization/pseudonymization for test data generation etc). Falcon
framework can maintain the common recipes and document their contract. We
could have a recipe repository that is pluggable, wherein user can submit
their recipes with its jars. This also allows user to create high-level
recipes using available ones.


On Fri, Aug 8, 2014 at 12:32 PM, Shwetha GS <shwetha.gs@inmobi.com> wrote:

> +1 for the feature
>
> 1. Is creation of recipe in user space or would that need to be bundled in
> falcon as part of distribution ? How do users provide support artifacts for
> the recipe
> We can have some in-built recipes, but we should also support recipes at
> user space
>
> 2. How do users track execution of instances of a particular recipe. Am
> assuming you can create multiple instances of a single recipe
> We can let the user specify process name
>
>
>
> On Fri, Aug 8, 2014 at 10:55 AM, Srikanth Sundarrajan <sriksun@hotmail.com
> >
> wrote:
>
> > +1 for the capability.
> >
> > Some clarifications.
> >
> > 1. Is creation of recipe in user space or would that need to be bundled
> in
> > falcon as part of distribution ? How do users provide support artifacts
> for
> > the recipe
> > 2. How do users track execution of instances of a particular recipe. Am
> > assuming you can create multiple instances of a single recipe
> > 3. Would simple properties table suffice?
> >
> > Regards
> > Srikanth Sundarrajan
> >
> > > Date: Thu, 7 Aug 2014 19:34:13 -0700
> > > Subject: [DISCUSS] Recipes in Falcon
> > > From: venkatesh@innerzeal.com
> > > To: dev@falcon.incubator.apache.org
> > >
> > > Hi folks,
> > >
> > > Falcon offers many services OOTB and caters to a wide array of use
> cases.
> > > However, there has been many asks that does not fit the functionality
> > > offered by Falcon. I'm proposing that we add recipes to Falcon which is
> > > similar to recipes in Whirr and other management solutions such as
> puppet
> > > and chef.
> > >
> > > Overview:
> > > A recipe essentially is a static process template with parameterized
> > > workflow to realize a specific use case. For example:
> > >
> > > * replicating directories from one HDFS cluster to another (not timed
> > > partitions)
> > > * replicating hive metadata (database, table, views, etc.)
> > > * replicating between HDFS and Hive - either way
> > > * anonymization of data based on schema
> > > * data masking
> > > * etc.
> > >
> > > Proposal:
> > > Falcon provides a Process abstraction that encapsulates the
> configuration
> > > for a user workflow with scheduling controls. All recipes can be
> modeled
> > > as a Process with in Falcon which executes the user workflow
> > > periodically. The process and its associated workflow are
> parameterized.
> > > The user will provide a properties file with name value pairs that are
> > > substituted by falcon before scheduling it.
> > >
> > > The CLI would look something like this:
> > > falcon -recipe $recipe_name -properties $properties_file
> > >
> > > Thoughts?
> > >
> > > --
> > > Regards,
> > > Venkatesh
> > >
> > > “Perfection (in design) is achieved not when there is nothing more to
> > add,
> > > but rather when there is nothing more to take away.”
> > > - Antoine de Saint-Exupéry
> >
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>

-- 
_____________________________________________________________
The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 
receipt.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message