Mailing-List: contact general-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of acm@hortonworks.com designates
 74.125.82.172 as permitted sender)
References: 
 <CADtX4R8dm8S0qx4ShCgn13UfaR1=5Mf6sci2bnbdy9AhDJN76g@mail.gmail.com>
From: Arun Murthy <acm@hortonworks.com>
In-Reply-To: 
 <CADtX4R8dm8S0qx4ShCgn13UfaR1=5Mf6sci2bnbdy9AhDJN76g@mail.gmail.com>
Mime-Version: 1.0 (1.0)
Date: Thu, 21 Mar 2013 08:49:24 +0100
Message-ID: <4906768017502491488@unknownmsgid>
Subject: Re: [VOTE] Accept Falcon into the Apache Incubator (was originally
 named Ivory)
To: "general@incubator.apache.org" <general@incubator.apache.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

+1 (binding)

Arun

On Mar 21, 2013, at 5:55 AM, Srikanth Sundarrajan
<srikanth.sundarrajan@inmobi.com> wrote:

> Hi,
>
> Thanks for participating in the proposal discussion on Falcon
> (formerly Ivory). I'd like to call a VOTE for acceptance of Apache
> Falcon into the Incubator. I'll let the vote run till (Tue 3/26 6pm IST).
>
> [ ]  +1 Accept Apache Falcon into the Incubator
> [ ]  +0 Don't care.
> [ ]  -1 Don't accept Apache Falcon into the Incubator because...
>
> Full proposal is pasted at the bottom of this email, and the
> corresponding wiki is http://wiki.apache.org/incubator/FalconProposal.
>
>
> Only VOTEs from Incubator PMC members are binding, but all are welcome
> to express their thoughts.
>
> Thanks,
> Srikanth Sundarrajan
> =3D Falcon Proposal =3D
>
> =3D=3D Abstract =3D=3D
> Falcon is a data processing and management solution for Hadoop
> designed for data motion, coordination of data pipelines, lifecycle
> management, and data discovery. Falcon enables end consumers to
> quickly onboard their data and its associated processing and
> management tasks on Hadoop clusters.
>
> =3D=3D Proposal =3D=3D
> Falcon will enable easy data management via declarative mechanism for
> Hadoop. Users of Falcon platform simply define infrastructure
> endpoints, data sets and processing rules declaratively. These
> declarative configurations are expressed in such a way that the
> dependencies between these configured entities are explicitly
> described. This information about inter-dependencies between various
> entities allows Falcon to orchestrate and manage various data
> management functions.
>
> The key use cases that Falcon addresses are:
> * Data Motion
> * Process orchestration and scheduling
> * Policy-based Lifecycle Management
> * Data Discovery
> * Operability/Usability
>
> With these features it is possible for users to onboard their data
> sets with a comprehensive and holistic understanding of how, when and
> where their data is managed across its lifecycle. Complex functions
> such as retrying failures, identifying possible SLA breaches or
> automated handling of input data changes are now simple directives.
> All the administrative functions and user level functions are
> available via RESTful APIs. CLI is simply a wrapper over the RESTful
> APIs.
>
> =3D=3D Background =3D=3D
> Hadoop and its ecosystem of products have made storing and processing
> massive amounts of data commonplace. This has enabled numerous
> organizations to gain valuable insights that they never could have
> achieved in the past. While it is easy to leverage Hadoop for
> crunching large volumes of data, organizing data, managing life cycle
> of data and processing data is fairly involved. This is solved
> adequately well in a classic data platform involving data warehouses
> and standard ETL (extract-transform-load) tools, but remains largely
> unsolved today. In addition to data processing complexities, Hadoop
> presents new sets of challenges and opportunities relating to
> management of data.
>
> Data Management on Hadoop encompasses data motion, process
> orchestration, lifecycle management, data discovery, etc. among other
> concerns that are beyond ETL. Falcon is a new data processing and
> management platform for Hadoop that solves this problem and creates
> additional opportunities by building on existing components within the
> Hadoop ecosystem (ex. Apache Oozie, Apache Hadoop DistCp etc.) without
> reinventing the wheel. Falcon has been in production at InMobi, going
> on its second year and has been managing hundreds of feeds and
> processes.
>
> Falcon is being developed by engineers employed with InMobi and
> Hortonworks. This platform addition will increase the adoption of
> Apache Hadoop by driving data management tractable for end users. We
> are therefore proposing to make Falcon an Apache open source project.
>
> =3D=3D Rationale =3D=3D
> The Falcon project aims to improve the usability of Apache Hadoop. As
> a result Apache Hadoop will grow its community of users by increasing
> the places Hadoop can be utilized and the use cases it will solve. By
> developing Falcon in Apache we hope to gather a diverse community of
> contributors, helping to ensure that Falcon is deployable for a broad
> range of scenarios. Members of the Hadoop development community will
> be able to influence Falcon=92s roadmap, and contribute to it. We
> believe having Falcon as part of the Apache Hadoop ecosystem will be a
> great benefit to all of Hadoop's users.
>
> =3D=3D Current Status =3D=3D
> Falcon is widely deployed in production within InMobi and moving on to
> its second year. A version with a valuable set of features is
> developed by the list of initial committers and is hosted on github.
>
> =3D=3D=3D Meritocracy =3D=3D=3D
> Our intent with this incubator proposal is to start building a diverse
> developer community around Falcon following the Apache meritocracy
> model. We have wanted to make the project open source and encourage
> contributors from multiple organizations from the start. We plan to
> provide plenty of support to new developers and to quickly recruit
> those who make solid contributions to committer status.
>
> =3D=3D=3D Community =3D=3D=3D
> We are happy to report that the initial team already represents
> multiple organizations. We hope to extend the user and developer base
> further in the future and build a solid open source community around
> Falcon.
>
> =3D=3D=3D Core Developers =3D=3D=3D
> Falcon is currently being developed by three engineers from InMobi =96
> Srikanth Sunderrajan, Shwetha G S, and Shaik Idris, two Hortonworks
> employees =96 Sanjay Radia and Venkatesh Seetharam. In addition, Rohini
> Palaniswamy and Thiruvel Thirumoolan, were also involved in the
> initial design discussions. Srikanth, Shwetha and Shaik are the
> original developers. All the engineers have built two generations of
> Data Management on Hadoop, having deep expertise in Hadoop and are
> quite familiar with the Hadoop Ecosystem. Samarth Gupta & Rishu
> Mehrothra, both from InMobi have build the QA automation for Falcon.
>
> =3D=3D=3D Alignment =3D=3D=3D
> The ASF is a natural host for Falcon given that it is already the home
> of Hadoop, Pig, Knox, HCatalog, and other emerging =93big data=94 softwar=
e
> projects. Falcon has been designed to solve the data management
> challenges and opportunities of the Hadoop ecosystem family of
> products. Falcon fills the gap that Hadoop ecosystem has been lacking
> in the areas of data processing and data lifecycle management.
>
> =3D=3D Known Risks =3D=3D
>
> =3D=3D=3D Orphaned products & Reliance on Salaried Developers =3D=3D=3D
> The core developers plan to work full time on the project. There is
> very little risk of Falcon getting orphaned. Falcon is in use by
> companies we work for so the companies have an interest in its
> continued vitality.
>
> =3D=3D=3D Inexperience with Open Source =3D=3D=3D
> All of the core developers are active users and followers of open
> source. Srikanth Sundarrajan has been contributing patches to Apache
> Hadoop and Apache Oozie, Shwetha GS has been contributing patches to
> Apache Oozie.  Seetharam Venkatesh is a committer on Apache Knox.
> Sharad Agarwal, Amareshwari SR (also a Apache Hive PMC member) and
> Sanjay Radia are PMC members on Apache Hadoop.
>
> =3D=3D=3D Homogeneous Developers =3D=3D=3D
> The current core developers are from diverse set of organizations such
> as InMobi and Hortonworks. We expect to quickly establish a developer
> community that includes contributors from several corporations post
> incubation.
>
> =3D=3D=3D Reliance on Salaried Developers =3D=3D=3D
> Currently, most developers are paid to do work on Falcon but few are
> contributing in their spare time. However, once the project has a
> community built around it post incubation, we expect to get committers
> and developers from outside the current core developers.
>
> =3D=3D=3D Relationships with Other Apache Products =3D=3D=3D
> Falcon is going to be used by the users of Hadoop and the Hadoop
> ecosystem in general.
>
> =3D=3D=3D A Excessive Fascination with the Apache Brand =3D=3D=3D
> While we respect the reputation of the Apache brand and have no doubts
> that it will attract contributors and users, our interest is primarily
> to give Falcon a solid home as an open source project following an
> established development model. We have also given reasons in the
> Rationale and Alignment sections.
>
> =3D=3D Documentation =3D=3Dhttp://wiki.apache.org/incubator/FalconProposa=
l
>
> =3D=3D Initial Source =3D=3D
> The source is currently in github repository at:
> https://github.com/sriksun/Falcon
>
> =3D=3D Source and Intellectual Property Submission Plan =3D=3D
> The complete Falcon code is under Apache Software License 2.
>
> =3D=3D External Dependencies =3D=3D
> The dependencies all have Apache compatible licenses. These include
> BSD, MIT licensed dependencies.
>
> =3D=3D Cryptography =3D=3D
> None
>
> =3D=3D Required Resources =3D=3D
>
> =3D=3D=3D Mailing lists =3D=3D=3D
>
> * falcon-dev AT incubator DOT apache DOT org
> * falcon-commits AT incubator DOT apache DOT org
> * falcon-user AT incubator apache DOT org
> * falcon-private AT incubator DOT apache DOT org
>
> =3D=3D=3D Subversion Directory =3D=3D=3D
> Git is the preferred source control system: git://git.apache.org/falcon
>
> =3D=3D=3D Issue Tracking =3D=3D=3D
> JIRA FALCON
>
> =3D=3D Initial Committers =3D=3D
> * Srikanth Sundarrajan (Srikanth.Sundarrajan AT inmobi DOT com)
> * Shwetha GS (shwetha.gs AT inmobi DOT com)
> * Shaik Idris (shaik.idris AT inmobi DOT com)
> * Venkatesh Seetharam (Venkatesh AT apache DOT org)
> * Sanjay Radia (sanjay AT apache DOT org)
> * Sharad Agarwal (sharad AT apache DOT org)
> * Amareshwari SR (amareshwari AT apache DOT org)
> * Samarth Gupta (samarth.gupta AT inmobi DOT com)
> * Rishu Mehrothra (rishu.mehrothra AT inmobi DOT com)
>
> =3D=3D Affiliations =3D=3D
> * Srikanth Sundarrajan (InMobi)
> * Shwetha GS (InMobi)
> * Shaik Idris (InMobi)
> * Venkatesh Seetharam (Hortonworks Inc.)
> * Sanjay Radia (Hortonworks Inc.)
> * Sharad Agarwal (InMobi)
> * Amareshwari SR (InMobi)
> * Samarth Gupta (InMobi)
> * Rishu Mehrothra (InMobi)
>
> =3D=3D Sponsors =3D=3D
>
> =3D=3D=3D Champion =3D=3D=3D
> * Arun C Murthy (acmurthy at apache dot org)
>
> =3D=3D=3D Nominated Mentors =3D=3D=3D
> * Alan Gates (gates AT apache DOT org)
> * Chris Douglas (cdouglas AT apache DOT org)
> * Devaraj  Das (ddas AT apache DOT org)
> * Owen O=92Malley (omalley AT apache DOT org)
>
> =3D=3D=3D Sponsoring Entity =3D=3D=3D
> Incubator PMC
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for th=
e
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileg=
ed
> information. If you are not the intended recipient you are hereby notifie=
d
> that any disclosure, copying, distribution or taking any action in relian=
ce
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmissi=
on
> of the information contained in this communication nor for any delay in i=
ts
> receipt.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org