incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Update of "FalconProposal" by ArunMurthy
Date Wed, 20 Mar 2013 20:24:42 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "FalconProposal" page has been changed by ArunMurthy:
http://wiki.apache.org/incubator/FalconProposal?action=diff&rev1=5&rev2=6

Comment:
Project Ivory renamed to Falcon

  ## page was renamed from IvoryProposal
- = Ivory Proposal =
+ = Falcon Proposal =
  
  == Abstract ==
- Ivory is a data processing and management solution for Hadoop designed for data motion,
coordination of data pipelines, lifecycle management, and data discovery. Ivory enables end
consumers to quickly onboard their data and its associated processing and management tasks
on Hadoop clusters.
+ Falcon is a data processing and management solution for Hadoop designed for data motion,
coordination of data pipelines, lifecycle management, and data discovery. Falcon enables end
consumers to quickly onboard their data and its associated processing and management tasks
on Hadoop clusters.
  
  == Proposal ==
- Ivory will enable easy data management via declarative mechanism for Hadoop. Users of Ivory
platform simply define infrastructure endpoints, data sets and processing rules declaratively.
These declarative configurations are expressed in such a way that the dependencies between
these configured entities are explicitly described. This information about inter-dependencies
between various entities allows Ivory to orchestrate and manage various data management functions.
+ Falcon will enable easy data management via declarative mechanism for Hadoop. Users of Falcon
platform simply define infrastructure endpoints, data sets and processing rules declaratively.
These declarative configurations are expressed in such a way that the dependencies between
these configured entities are explicitly described. This information about inter-dependencies
between various entities allows Falcon to orchestrate and manage various data management functions.
  
- The key use cases that Ivory addresses are:
+ The key use cases that Falcon addresses are:
   * Data Motion
   * Process orchestration and scheduling
   * Policy-based Lifecycle Management
@@ -21, +21 @@

  == Background ==
  Hadoop and its ecosystem of products have made storing and processing massive amounts of
data commonplace. This has enabled numerous organizations to gain valuable insights that they
never could have achieved in the past. While it is easy to leverage Hadoop for crunching large
volumes of data, organizing data, managing life cycle of data and processing data is fairly
involved. This is solved adequately well in a classic data platform involving data warehouses
and standard ETL (extract-transform-load) tools, but remains largely unsolved today. In addition
to data processing complexities, Hadoop presents new sets of challenges and opportunities
relating to management of data.
  
- Data Management on Hadoop encompasses data motion, process orchestration, lifecycle management,
data discovery, etc. among other concerns that are beyond ETL. Ivory is a new data processing
and management platform for Hadoop that solves this problem and creates additional opportunities
by building on existing components within the Hadoop ecosystem (ex. Apache Oozie, Apache Hadoop
DistCp etc.) without reinventing the wheel. Ivory has been in production at InMobi, going
on its second year and has been managing hundreds of feeds and processes.
+ Data Management on Hadoop encompasses data motion, process orchestration, lifecycle management,
data discovery, etc. among other concerns that are beyond ETL. Falcon is a new data processing
and management platform for Hadoop that solves this problem and creates additional opportunities
by building on existing components within the Hadoop ecosystem (ex. Apache Oozie, Apache Hadoop
DistCp etc.) without reinventing the wheel. Falcon has been in production at InMobi, going
on its second year and has been managing hundreds of feeds and processes.
   
- Ivory is being developed by engineers employed with InMobi and Hortonworks. This platform
addition will increase the adoption of Apache Hadoop by driving data management tractable
for end users. We are therefore proposing to make Ivory an Apache open source project.
+ Falcon is being developed by engineers employed with InMobi and Hortonworks. This platform
addition will increase the adoption of Apache Hadoop by driving data management tractable
for end users. We are therefore proposing to make Falcon an Apache open source project.
  
  == Rationale ==
- The Ivory project aims to improve the usability of Apache Hadoop. As a result Apache Hadoop
will grow its community of users by increasing the places Hadoop can be utilized and the use
cases it will solve. By developing Ivory in Apache we hope to gather a diverse community of
contributors, helping to ensure that Ivory is deployable for a broad range of scenarios. Members
of the Hadoop development community will be able to influence Ivory’s roadmap, and contribute
to it. We believe having Ivory as part of the Apache Hadoop ecosystem will be a great benefit
to all of Hadoop's users.
+ The Falcon project aims to improve the usability of Apache Hadoop. As a result Apache Hadoop
will grow its community of users by increasing the places Hadoop can be utilized and the use
cases it will solve. By developing Falcon in Apache we hope to gather a diverse community
of contributors, helping to ensure that Falcon is deployable for a broad range of scenarios.
Members of the Hadoop development community will be able to influence Falcon’s roadmap,
and contribute to it. We believe having Falcon as part of the Apache Hadoop ecosystem will
be a great benefit to all of Hadoop's users.
  
  == Current Status ==
- Ivory is widely deployed in production within InMobi and moving on to its second year. A
version with a valuable set of features is developed by the list of initial committers and
is hosted on github.
+ Falcon is widely deployed in production within InMobi and moving on to its second year.
A version with a valuable set of features is developed by the list of initial committers and
is hosted on github.
  
  === Meritocracy ===
- Our intent with this incubator proposal is to start building a diverse developer community
around Ivory following the Apache meritocracy model. We have wanted to make the project open
source and encourage contributors from multiple organizations from the start. We plan to provide
plenty of support to new developers and to quickly recruit those who make solid contributions
to committer status.
+ Our intent with this incubator proposal is to start building a diverse developer community
around Falcon following the Apache meritocracy model. We have wanted to make the project open
source and encourage contributors from multiple organizations from the start. We plan to provide
plenty of support to new developers and to quickly recruit those who make solid contributions
to committer status.
  
  === Community ===
- We are happy to report that the initial team already represents multiple organizations.
We hope to extend the user and developer base further in the future and build a solid open
source community around Ivory.
+ We are happy to report that the initial team already represents multiple organizations.
We hope to extend the user and developer base further in the future and build a solid open
source community around Falcon.
  
  === Core Developers ===
- Ivory is currently being developed by three engineers from InMobi – Srikanth Sunderrajan,
Shwetha G S, and Shaik Idris, two Hortonworks employees – Sanjay Radia and Venkatesh Seetharam.
In addition, Rohini Palaniswamy and Thiruvel Thirumoolan, were also involved in the initial
design discussions. Srikanth, Shwetha and Shaik are the original developers. All the engineers
have built two generations of Data Management on Hadoop, having deep expertise in Hadoop and
are quite familiar with the Hadoop Ecosystem. Samarth Gupta & Rishu Mehrothra, both from
InMobi have build the QA automation for Ivory.
+ Falcon is currently being developed by three engineers from InMobi – Srikanth Sunderrajan,
Shwetha G S, and Shaik Idris, two Hortonworks employees – Sanjay Radia and Venkatesh Seetharam.
In addition, Rohini Palaniswamy and Thiruvel Thirumoolan, were also involved in the initial
design discussions. Srikanth, Shwetha and Shaik are the original developers. All the engineers
have built two generations of Data Management on Hadoop, having deep expertise in Hadoop and
are quite familiar with the Hadoop Ecosystem. Samarth Gupta & Rishu Mehrothra, both from
InMobi have build the QA automation for Falcon.
  
  === Alignment ===
- The ASF is a natural host for Ivory given that it is already the home of Hadoop, Pig, Knox,
HCatalog, and other emerging “big data” software projects. Ivory has been designed to
solve the data management challenges and opportunities of the Hadoop ecosystem family of products.
Ivory fills the gap that Hadoop ecosystem has been lacking in the areas of data processing
and data lifecycle management.
+ The ASF is a natural host for Falcon given that it is already the home of Hadoop, Pig, Knox,
HCatalog, and other emerging “big data” software projects. Falcon has been designed to
solve the data management challenges and opportunities of the Hadoop ecosystem family of products.
Falcon fills the gap that Hadoop ecosystem has been lacking in the areas of data processing
and data lifecycle management.
  
  == Known Risks ==
  
  === Orphaned products & Reliance on Salaried Developers ===
- The core developers plan to work full time on the project. There is very little risk of
Ivory getting orphaned. Ivory is in use by companies we work for so the companies have an
interest in its continued vitality.
+ The core developers plan to work full time on the project. There is very little risk of
Falcon getting orphaned. Falcon is in use by companies we work for so the companies have an
interest in its continued vitality.
  
  === Inexperience with Open Source ===
  All of the core developers are active users and followers of open source. Srikanth Sundarrajan
has been contributing patches to Apache Hadoop and Apache Oozie, Shwetha GS has been contributing
patches to Apache Oozie.  Seetharam Venkatesh is a committer on Apache Knox. Sharad Agarwal,
Amareshwari SR (also a Apache Hive PMC member) and Sanjay Radia are PMC members on Apache
Hadoop.
@@ -55, +55 @@

  The current core developers are from diverse set of organizations such as InMobi and Hortonworks.
We expect to quickly establish a developer community that includes contributors from several
corporations post incubation.
  
  === Reliance on Salaried Developers ===
- Currently, most developers are paid to do work on Ivory but few are contributing in their
spare time. However, once the project has a community built around it post incubation, we
expect to get committers and developers from outside the current core developers.
+ Currently, most developers are paid to do work on Falcon but few are contributing in their
spare time. However, once the project has a community built around it post incubation, we
expect to get committers and developers from outside the current core developers.
  
  === Relationships with Other Apache Products ===
- Ivory is going to be used by the users of Hadoop and the Hadoop ecosystem in general.
+ Falcon is going to be used by the users of Hadoop and the Hadoop ecosystem in general.
  
  === A Excessive Fascination with the Apache Brand ===
- While we respect the reputation of the Apache brand and have no doubts that it will attract
contributors and users, our interest is primarily to give Ivory a solid home as an open source
project following an established development model. We have also given reasons in the Rationale
and Alignment sections.
+ While we respect the reputation of the Apache brand and have no doubts that it will attract
contributors and users, our interest is primarily to give Falcon a solid home as an open source
project following an established development model. We have also given reasons in the Rationale
and Alignment sections.
  
  == Documentation ==
- http://wiki.apache.org/incubator/IvoryProposal
+ http://wiki.apache.org/incubator/FalconProposal
  
  == Initial Source ==
- The source is currently in github repository at: https://github.com/sriksun/Ivory
+ The source is currently in github repository at: https://github.com/sriksun/Falcon
  
  == Source and Intellectual Property Submission Plan ==
- The complete Ivory code is under Apache Software License 2.
+ The complete Falcon code is under Apache Software License 2.
  
  == External Dependencies ==
  The dependencies all have Apache compatible licenses. These include BSD, MIT licensed dependencies.
@@ -82, +82 @@

  
  === Mailing lists ===
  
-  * ivory-dev AT incubator DOT apache DOT org
+  * falcon-dev AT incubator DOT apache DOT org
-  * ivory-commits AT incubator DOT apache DOT org
+  * falcon-commits AT incubator DOT apache DOT org
-  * ivory-user AT incubator apache DOT org
+  * falcon-user AT incubator apache DOT org
-  * ivory-private AT incubator DOT apache DOT org
+  * falcon-private AT incubator DOT apache DOT org
  
  === Subversion Directory ===
- Git is the preferred source control system: git://git.apache.org/ivory
+ Git is the preferred source control system: git://git.apache.org/falcon
  
  === Issue Tracking ===
- JIRA IVORY
+ JIRA FALCON
  
  == Initial Committers ==
   * Srikanth Sundarrajan (Srikanth.Sundarrajan AT inmobi DOT com)

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message