From general-return-30271-apmail-incubator-general-archive=incubator.apache.org@incubator.apache.org Fri Jun 17 08:43:18 2011 Return-Path: X-Original-To: apmail-incubator-general-archive@www.apache.org Delivered-To: apmail-incubator-general-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 40FC244D1 for ; Fri, 17 Jun 2011 08:43:18 +0000 (UTC) Received: (qmail 73195 invoked by uid 500); 17 Jun 2011 08:43:17 -0000 Delivered-To: apmail-incubator-general-archive@incubator.apache.org Received: (qmail 72978 invoked by uid 500); 17 Jun 2011 08:43:17 -0000 Mailing-List: contact general-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@incubator.apache.org Delivered-To: mailing list general@incubator.apache.org Delivered-To: moderator for general@incubator.apache.org Received: (qmail 88087 invoked by uid 99); 16 Jun 2011 20:15:28 -0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted nor denied by domain of gates@yahoo-inc.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com; s=cobra; t=1308255260; bh=3KZVrYu7LLuYqUPNGzx2so8U1dO94Htg983xHeiUU4M=; h=Message-Id:From:To:In-Reply-To:Content-Type: Content-Transfer-Encoding:Mime-Version:Subject:Date:References; b=RmpDSgn3H2mVBJ2aXbuHi1Mn3wV/kTLhqlpD9APvfl4+QdSNNJeLDFk4oe8Vo7q28 0W/ttrHpTHGWB8gs131nGWmF3/Oe+5KeLfgUhCyNKHaeFAjJcglYE6v+RbYDzqUUea Y9+SIdjkQVio7ZlUOFPM+Vru4+1gOV5ffaj21L7w= Message-Id: From: Alan Gates To: "general@incubator.apache.org" In-Reply-To: Content-Type: text/plain; charset=WINDOWS-1252; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v936) Subject: Re: [PROPOSAL] Bigtop for the Apache Incubator Date: Thu, 16 Jun 2011 13:14:19 -0700 References: X-Mailer: Apple Mail (2.936) X-Virus-Checked: Checked by ClamAV on apache.org Tom, I'd like to join as a committer and mentor as well. Alan. On Jun 13, 2011, at 9:26 PM, Tom White wrote: > Hi, > > I would like to propose Bigtop to be an Apache Incubator project. > Bigtop is a project for the development of packaging and tests of the > Hadoop ecosystem. The goal is to do testing at various levels > (packaging, platform, runtime, upgrade, etc...) developed by a > community with a focus on the system as a whole, rather than > individual projects. > > Here's a link to the proposal on the wiki > http://wiki.apache.org/incubator/BigtopProposal > > I've also included the initial contents below. > > Cheers, > Tom > > =3D Bigtop - Apache Hadoop Ecosystem Packaging and Test =3D > > =3D=3D Abstract =3D=3D > > Bigtop - a project for the development of packaging and tests of the > Hadoop ecosystem. > > =3D=3D Proposal =3D=3D > > The primary goal of Bigtop is to build a community around the > packaging and interoperability testing of Hadoop-related projects. > This includes testing at various levels (packaging, platform, runtime, > upgrade, etc...) developed by a community with a focus on the system > as a whole, rather than individual projects. > > Build, packaging and integration test code that depends upon official > releases of the Apache Hadoop-related projects (HDFS, MapReduce, > HBase, Hive, Pig, ZooKeeper, etc...) will be developed and released by > this project. As bugs and other issues are found we expect these to be > fixed upstream. > > =3D=3D Background =3D=3D > > The initial packaging and test code for Bigtop was developed by > Cloudera to package projects from the Apache Hadoop ecosystem and > provide a consistent, inter-operable framework. > > =3D=3D Rationale =3D=3D > > Hadoop defines itself as: > > {{{ > The Apache Hadoop project develops open-source software for reliable, > scalable, distributed computing. Hadoop includes these subprojects: > > * Hadoop Common: The common utilities that support the other Hadoop =20= > subprojects. > * HDFS: A distributed file system that provides high throughput access > to application data. > * MapReduce: A software framework for distributed processing of large > data sets on compute clusters. > }}} > > There are also several other Hadoop-related projects at Apache. Some > TLP examples include HBase, Hive, Mahout, ZooKeeper, and Pig. There > are also several new projects in the Incubator such as HCatalog and > Sqoop. > > There is limited interoperability testing done the projects > themselves. The intent of this project is to build a community where > the projects are brought together, packaged, and tested for > interoperability. > > Projects such as Apache Whirr (incubating), which deploy and use a > collection of Hadoop-related projects, would benefit from the > interoperability testing done by Bigtop, rather than picking and > testing project combinations themselves. > > =3D=3D Initial Goals =3D=3D > > Much of the code for Bigtop has been released by Cloudera under the > Apache 2.0 license for over two years. > > Some current goals include: > * create a set of packages for the Hadoop ecosystem, over a wide > range of platforms > * interoperability test these projects > * document project sets that are known to work well together > > Bigtop=92s release artifact would consist of a single tarball of > packaging and test code that, when built, would produce source and > binary Linux packages for the upstream projects. > > =3D Current Status =3D > > =3D=3D Meritocracy =3D=3D > > Bigtop was originally developed and released as an open source > packaging infrastructure, CDH, by Cloudera. > > =3D=3D Community =3D=3D > > The community is primarily the original developers at Cloudera, > however a number of contributions to the packaging specifications have > been accepted from outside contributors. Growing a diverse community > is the main reason to bring Bigtop to the Apache Incubator. > > =3D=3D Core Developers =3D=3D > > The core developers for Bigtop project are: > * Andrew Bayer has extensive expertise with build tools, specifically > Jenkins continuous integration and Maven. > * Peter Linnell has contributed to the RPM packaging. > * Bruno Mah=E9 has overseen much of the development of the RPM and > Debian packaging system. > * Roman Shaposhnik designed and implemented the system testing =20 > framework. > > Many of the committers to the Bigtop project have contributed towards > Hadoop or related Apache projects (Alejandro Abdelnur, Eli Collins, > Patrick Hunt, John Sichi, Michael Stack, Tom White) and are familiar > with Apache principals and philosophy for community driven software > development. > > =3D=3D Alignment =3D=3D > > We expect projects in Bigtop to be drawn from Hadoop and related > projects at Apache. Bigtop will complement these projects (Hadoop, > Pig, Hive, HBase, etc...) by providing an environment for contributors > interested in building more complex data processing pipelines to work > together integrating more than a single project into a well tested > whole. > > =3D Known Risks =3D > > =3D=3D Orphaned Products =3D=3D > > The contributors are leading vendors of Hadoop-based technologies and > have a long standing in the Hadoop community. There is minimal risk of > this work becoming non-strategic and the contributors are confident > that a larger community will form within the project in a relatively > short space of time. > > =3D=3D Inexperience with Open Source =3D=3D > > All code developed for Bigtop has been open sourced under the Apache > 2.0 license. Most committers of Bigtop project are intimately familiar > with the Apache model for open-source development and are experienced > with working with new contributors. > > =3D=3D Homogeneous Developers =3D=3D > > The initial set of committers is from a small set of organizations and > numerous existing Apache projects. We expect that once approved for > incubation, the project will attract new contributors from more > organizations and will thus grow organically. > > =3D=3D Reliance on Salaried Developers =3D=3D > > It is expected that Bigtop will be developed on salaried and volunteer > time, although all of the initial developers will work on it mainly on > salaried time. > > =3D=3D Relationships with Other Apache Products =3D=3D > > Bigtop depends upon other Apache Projects including Apache Hadoop, > Apache HBase, Apache Hive, Apache Pig, Apache Zookeeper, Apache > Thrift, Apache Avro. The build system uses Apache Ant and Apache > Maven. > > =3D=3D An Excessive Fascination with the Apache Brand =3D=3D > > We would like Bigtop to become an Apache project to further foster a > healthy community of contributors and consumers around > interoperability, testing and packaging of Hadoop projects. Since > Bigtop directly interacts with many Apache Hadoop-related projects and > solves important problems of many Hadoop users, residing in the the > Apache Software Foundation will increase interaction with the larger > community. > > =3D Documentation =3D > > * Bigtop will develop its own documentation detailing how to build, > test, install, configure and debug. > > =3D Initial Source =3D > > * https://github.com/cloudera/bigtop > > =3D=3D Source and Intellectual Property Submission Plan =3D=3D > > * The initial source is already licensed under the Apache License, =20 > Version 2.0. > > https://github.com/cloudera/bigtop > > =3D=3D External Dependencies =3D=3D > > The required external dependencies are all Apache License or > compatible licenses. > > =3D=3D Cryptography =3D=3D > > Bigtop doesn't use cryptography itself, however Hadoop projects use > standard APIs and tools for SSH and SSL communication where necessary. > > =3D Required Resources =3D > > =3D=3D Mailing lists =3D=3D > > * bigtop-private (with moderated subscriptions) > * bigtop-dev > * bigtop-commits > * bigtop-user > > =3D=3D Subversion Directory =3D=3D > > https://svn.apache.org/repos/asf/incubator/bigtop > > =3D=3D Issue Tracking =3D=3D > > JIRA BIGTOP (Bigtop) > > =3D=3D Other Resources =3D=3D > > The existing code already has unit and integration tests so we would > like a Jenkins instance to run them whenever a new patch is submitted. > This can be added after project creation. > > =3D Initial Committers =3D > > * Alejandro Abdelnur (tucu at cloudera dot com) > * Andrew Bayer (abayer at cloudera dot com) > * Eli Collins (eli at apache dot org) > * Travis Crawford (travis at twitter dot com) > * Bruno Mah=E9 (bruno at cloudera dot com) > * Patrick Hunt (phunt at apache dot org) > * Peter Linnell (plinnell at cloudera dot com) > * James Page (James.page at canonical dot com) > * Roman Shaposhnik (rvs at cloudera dot com) > * John Sichi (jvs at apache dot org) > * Michael Stack (stack at apache dot org) > * Tom White (tomwhite at apache dot org) > > =3D Affiliations =3D > > * Alejandro Abdelnur, Cloudera > * Andrew Bayer, Cloudera > * Eli Collins, Cloudera > * Travis Crawford, Twitter > * Bruno Mah=E9, Cloudera > * Patrick Hunt, Cloudera > * Peter Linnell, Cloudera > * James Page, Canonical > * Roman Shaposhnik, Cloudera > * John Sichi, Facebook > * Michael Stack, StumbleUpon > * Tom White, Cloudera > > =3D Sponsors =3D > > =3D=3D Champion =3D=3D > > * Patrick Hunt > > =3D=3D Nominated Mentors =3D=3D > > * Patrick Hunt > * Tom White > > =3D=3D Sponsoring Entity =3D=3D > > * Apache Incubator PMC > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org > For additional commands, e-mail: general-help@incubator.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.org