incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Update of "BigtopProposal" by SteveLoughran
Date Tue, 14 Jun 2011 11:46:14 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "BigtopProposal" page has been changed by SteveLoughran:
http://wiki.apache.org/incubator/BigtopProposal?action=diff&rev1=3&rev2=4

Comment:
more on 

  * MapReduce: A software framework for distributed processing of large data sets on compute
clusters.
  }}}
  
- There are also several other Hadoop-related projects at Apache.  Some TLP examples include
HBase, Hive, Mahout, ZooKeeper, and Pig.  There are also several new projects in the Incubator
such as HCatalog and Sqoop.
+ There are also several other Hadoop-related projects at Apache.  Some TLP examples include
HBase, Hive, Mahout, ZooKeeper, and Pig.  There are also several new projects in the Incubator
such as HCatalog, Hama and Sqoop.
  
+ From a packaging and deployment perspective, the current loosely-coupled nature of the project
has limitations:
+  1. Insufficient building against trunk versions of dependent projects (in the style of
Apache Gump).
+  1. Insufficient testing against the trunk versions of dependent projects.
+  1. No consistent packaging for the Linux servers which provide the main Hadoop datacenter
platform.
+  1. No functional testing against multi-machine clusters as part of the regular automated
build process. This is due to a lack of a physical or virtual Hadoop cluster for testing,
and not enough test suites designed to run against a live cluster with known datasets.
+ 
- There is limited interoperability testing done the projects themselves. The intent of this
project is to build a community where the projects are brought together, packaged, and tested
for interoperability.
+ The intent of this project is to build a community where the projects are brought together,
packaged, and tested for interoperability.
  
  Projects such as Apache Whirr (incubating), which deploy and use a collection of Hadoop-related
projects, would benefit from the interoperability testing done by Bigtop, rather than picking
and testing project combinations themselves.
  
@@ -63, +69 @@

   * Bruno Mahé has overseen much of the development of the RPM and Debian packaging system.
   * Roman Shaposhnik designed and implemented the system testing framework.
  
- Many of the committers to the Bigtop project have contributed towards Hadoop or related
Apache projects (Alejandro Abdelnur, Eli Collins, Patrick Hunt, John Sichi, Michael Stack,
Tom White) and are familiar with Apache principals and philosophy for community driven software
development.
+ Many of the committers to the Bigtop project have contributed towards Hadoop or related
Apache projects (Alejandro Abdelnur, Eli Collins, Patrick Hunt, Steve Loughran, John Sichi,
Michael Stack, Tom White) and are familiar with Apache principals and philosophy for community
driven software development. 
  
  == Alignment ==
  
- We expect projects in Bigtop to be drawn from Hadoop and related projects at Apache. Bigtop
will complement these projects (Hadoop, Pig, Hive, HBase, etc...) by providing an environment
for contributors interested in building more complex data processing pipelines to work together
integrating more than a single project into a well tested whole.
+ We expect projects in Bigtop to be drawn from Hadoop and related projects at Apache. Bigtop
will complement these projects (Hadoop, Pig, Hive, HBase, etc...) by providing an environment
for contributors interested in building more complex data processing pipelines to work together
integrating more than a single project into a well-tested whole.
  
  = Known Risks =
  
@@ -90, +96 @@

  == Relationships with Other Apache Products ==
  
  Bigtop depends upon other Apache Projects including Apache Hadoop, Apache HBase, Apache
Hive, Apache Pig, Apache Zookeeper, Apache Thrift, Apache Avro, Apache Whirr. The build system
uses Apache Ant and Apache Maven.
+ 
+ One consequence of a unified packaging and release process for the "Apache Hadoop stack"
is that it will encourage closer co-operation between the related Apache projects, and also
create pressure for synchronized releases of the artifacts. This is not necessarily a bad
thing -the issues with getting a stable append operation into the HDFS filesystem has created
problems for the HBase team, problems that synchronized releases would fix. However, it will
require the individual projects to care about their dependent projects, and to work together
on a more synchronous schedule for project releases. While ultimately this will be beneficial,
it may be problematic at first, and will require engagement from committers in the specific
Apache projects to address compatibility problems, and from the PMCs for coordinated releases.

  
  == An Excessive Fascination with the Apache Brand ==
  

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message