Return-Path: X-Original-To: apmail-incubator-general-archive@www.apache.org Delivered-To: apmail-incubator-general-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 23A726458 for ; Sat, 18 Jun 2011 03:06:03 +0000 (UTC) Received: (qmail 96701 invoked by uid 500); 18 Jun 2011 03:06:00 -0000 Delivered-To: apmail-incubator-general-archive@incubator.apache.org Received: (qmail 96511 invoked by uid 500); 18 Jun 2011 03:06:00 -0000 Mailing-List: contact general-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@incubator.apache.org Delivered-To: mailing list general@incubator.apache.org Received: (qmail 96494 invoked by uid 99); 18 Jun 2011 03:05:58 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 03:05:58 +0000 Received: from localhost (HELO [1.108.10.40]) (127.0.0.1) (smtp-auth username edwardyoon, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 03:05:57 +0000 Subject: Re: [VOTE] Accept Bigtop for incubation References: From: "Edward J. Yoon" Content-Type: multipart/alternative; boundary=Apple-Mail-1--653335474 X-Mailer: iPhone Mail (8J2) In-Reply-To: Message-Id: Date: Sat, 18 Jun 2011 12:05:39 +0900 To: "general@incubator.apache.org" Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (iPhone Mail 8J2) --Apple-Mail-1--653335474 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 +1 (non-binding) Sent from my iPhone On 2011. 6. 18., at =EC=98=A4=EC=A0=84 4:20, "Zimdars, Paul A (3880-Affiliat= e)" wrote: > +1 (non-binding) >=20 > Paul >=20 > On Jun 17, 2011, at 10:15 AM, "Tom White" wrote: >=20 >> As there are no active discussions on the proposal thread, I would >> like to initiate a vote to accept Bigtop as an Apache Incubator >> project. >>=20 >> The proposal is available at >>=20 >> http://wiki.apache.org/incubator/BigtopProposal?action=3Drecall&rev=3D13 >>=20 >> I've also put a copy of the proposal at the end of this email. >>=20 >> The discussion thread is available at >>=20 >> http://mail-archives.apache.org/mod_mbox/incubator-general/201106.mbox/%3= CBANLkTimriyVS5G5MAKLQviNAUZ9H6S5hWw@mail.gmail.com%3E >>=20 >> Please cast your votes: >>=20 >> [ ] +1 Accept Bigtop for incubation >> [ ] +0 Indifferent to Bigtop incubation >> [ ] -1 Reject Bigtop for incubation >>=20 >> This vote will close 72 hours from now. >>=20 >> Thanks, >> Tom >>=20 >> =3D Bigtop - Apache Hadoop Ecosystem Packaging and Test =3D >>=20 >> =3D=3D Abstract =3D=3D >>=20 >> Bigtop - a project for the development of packaging and tests of the >> Hadoop ecosystem. >>=20 >> =3D=3D Proposal =3D=3D >>=20 >> The primary goal of Bigtop is to build a community around the >> packaging and interoperability testing of Hadoop-related projects. >> This includes testing at various levels (packaging, platform, runtime, >> upgrade, etc...) developed by a community with a focus on the system >> as a whole, rather than individual projects. >>=20 >> Build, packaging and integration test code that depends upon official >> releases of the Apache Hadoop-related projects (HDFS, MapReduce, >> HBase, Hive, Pig, ZooKeeper, etc...) will be developed and released by >> this project. As bugs and other issues are found we expect these to be >> fixed upstream. >>=20 >> =3D=3D Background =3D=3D >>=20 >> The initial packaging and test code for Bigtop was developed by >> Cloudera to package projects from the Apache Hadoop ecosystem and >> provide a consistent, inter-operable framework. >>=20 >> =3D=3D Rationale =3D=3D >>=20 >> Hadoop defines itself as: >>=20 >> {{{ >> The Apache Hadoop project develops open-source software for reliable, >> scalable, distributed computing. Hadoop includes these subprojects: >>=20 >> * Hadoop Common: The common utilities that support the other Hadoop subpr= ojects. >> * HDFS: A distributed file system that provides high throughput access >> to application data. >> * MapReduce: A software framework for distributed processing of large >> data sets on compute clusters. >> }}} >>=20 >> There are also several other Hadoop-related projects at Apache. Some >> TLP examples include HBase, Hive, Mahout, ZooKeeper, and Pig. There >> are also several new projects in the Incubator such as HCatalog, Hama >> and Sqoop. >>=20 >> =46rom a packaging and deployment perspective, the current >> loosely-coupled nature of the project has limitations: >> 1. Insufficient building against trunk versions of dependent projects >> (in the style of Apache Gump). >> 1. Insufficient testing against the trunk versions of dependent projects.= >> 1. No consistent packaging for the Linux servers which provide the >> main Hadoop datacenter platform. >> 1. No functional testing against multi-machine clusters as part of >> the regular automated build process. This is due to a lack of a >> physical or virtual Hadoop cluster for testing, and not enough test >> suites designed to run against a live cluster with known datasets. >>=20 >> The intent of this project is to build a community where the projects >> are brought together, packaged, and tested for interoperability. >>=20 >> Projects such as Apache Whirr (incubating), which deploy and use a >> collection of Hadoop-related projects, would benefit from the >> interoperability testing done by Bigtop, rather than picking and >> testing project combinations themselves. >>=20 >> =3D=3D Initial Goals =3D=3D >>=20 >> Much of the code for Bigtop has been released by Cloudera under the >> Apache 2.0 license for over two years. >>=20 >> Some current goals include: >> * create a set of packages for the Hadoop ecosystem, over a wide >> range of platforms >> * interoperability test these projects >> * document project sets that are known to work well together >>=20 >> Bigtop=E2=80=99s release artifact would consist of a single tarball of >> packaging and test code that, when built, would produce source and >> binary Linux packages for the upstream projects. >>=20 >> =3D Current Status =3D >>=20 >> =3D=3D Meritocracy =3D=3D >>=20 >> Bigtop was originally developed and released as an open source >> packaging infrastructure, CDH, by Cloudera. >>=20 >> =3D=3D Community =3D=3D >>=20 >> The community is primarily the original developers at Cloudera, >> however a number of contributions to the packaging specifications have >> been accepted from outside contributors. Growing a diverse community >> is the main reason to bring Bigtop to the Apache Incubator. >>=20 >> =3D=3D Core Developers =3D=3D >>=20 >> The core developers for Bigtop project are: >> * Andrew Bayer has extensive expertise with build tools, specifically >> Jenkins continuous integration and Maven. >> * Peter Linnell has contributed to the RPM packaging. >> * Bruno Mah=C3=A9 has overseen much of the development of the RPM and >> Debian packaging system. >> * Roman Shaposhnik and Konstantin Boudnik designed and implemented >> the system testing framework. >>=20 >> Many of the committers to the Bigtop project have contributed towards >> Hadoop or related Apache projects (Alejandro Abdelnur, Konstantin >> Boudnik, Eli Collins, Alan Gates, Patrick Hunt, Steve Loughran, Owen >> O'Malley, John Sichi, Michael Stack, Tom White) and are familiar with >> Apache principals and philosophy for community driven software >> development. >>=20 >> =3D=3D Alignment =3D=3D >>=20 >> We expect projects in Bigtop to be drawn from Hadoop and related >> projects at Apache. Bigtop will complement these projects (Hadoop, >> Pig, Hive, HBase, etc...) by providing an environment for contributors >> interested in building more complex data processing pipelines to work >> together integrating more than a single project into a well-tested >> whole. >>=20 >> =3D Known Risks =3D >>=20 >> =3D=3D Orphaned Products =3D=3D >>=20 >> The contributors are leading vendors of Hadoop-based technologies and >> have a long standing in the Hadoop community. There is minimal risk of >> this work becoming non-strategic and the contributors are confident >> that a larger community will form within the project in a relatively >> short space of time. >>=20 >> =3D=3D Inexperience with Open Source =3D=3D >>=20 >> All code developed for Bigtop has been open sourced under the Apache >> 2.0 license. Most committers of Bigtop project are intimately familiar >> with the Apache model for open-source development and are experienced >> with working with new contributors. >>=20 >> =3D=3D Homogeneous Developers =3D=3D >>=20 >> The initial set of committers is from a small set of organizations and >> numerous existing Apache projects. We expect that once approved for >> incubation, the project will attract new contributors from more >> organizations and will thus grow organically. >>=20 >> =3D=3D Reliance on Salaried Developers =3D=3D >>=20 >> It is expected that Bigtop will be developed on salaried and volunteer >> time, although all of the initial developers will work on it mainly on >> salaried time. >>=20 >> =3D=3D Relationships with Other Apache Products =3D=3D >>=20 >> Bigtop depends upon other Apache Projects including Apache Hadoop, >> Apache HBase, Apache Hive, Apache Pig, Apache Zookeeper, Apache >> Thrift, Apache Avro, Apache Whirr. The build system uses Apache Ant >> and Apache Maven. >>=20 >> =3D=3D An Excessive Fascination with the Apache Brand =3D=3D >>=20 >> We would like Bigtop to become an Apache project to further foster a >> healthy community of contributors and consumers around >> interoperability, testing and packaging of Hadoop projects. Since >> Bigtop directly interacts with many Apache Hadoop-related projects and >> solves important problems of many Hadoop users, residing in the the >> Apache Software Foundation will increase interaction with the larger >> community. >>=20 >> =3D Documentation =3D >>=20 >> * Bigtop will develop its own documentation detailing how to build, >> test, install, configure and debug. >>=20 >> =3D Initial Source =3D >>=20 >> * https://github.com/cloudera/bigtop >>=20 >> =3D=3D Source and Intellectual Property Submission Plan =3D=3D >>=20 >> * The initial source is already licensed under the Apache License, Versio= n 2.0. >>=20 >> https://github.com/cloudera/bigtop >>=20 >> =3D=3D External Dependencies =3D=3D >>=20 >> The required external dependencies are all Apache License or >> compatible licenses. >>=20 >> =3D=3D Cryptography =3D=3D >>=20 >> Bigtop doesn't use cryptography itself, however Hadoop projects use >> standard APIs and tools for SSH and SSL communication where necessary. >>=20 >> =3D Required Resources =3D >>=20 >> =3D=3D Mailing lists =3D=3D >>=20 >> * bigtop-private (with moderated subscriptions) >> * bigtop-dev >> * bigtop-commits >> * bigtop-user >>=20 >> =3D=3D Subversion Directory =3D=3D >>=20 >> https://svn.apache.org/repos/asf/incubator/bigtop >>=20 >> =3D=3D Issue Tracking =3D=3D >>=20 >> JIRA BIGTOP (Bigtop) >>=20 >> =3D=3D Other Resources =3D=3D >>=20 >> The existing code already has unit and integration tests so we would >> like a Jenkins instance to run them whenever a new patch is submitted. >> This can be added after project creation. >>=20 >> To test RPM & deb install/uninstall and upgrade, it is useful to have >> a set of Virtual Machine images in known states, and servers that can >> bring them up. It should be possible to use Apache Whirr to >> choreograph the VM setup/teardown, so these tests could be performed >> against VMs on developer desktops or large scale VM-hosting platforms. >> For the latter, VM hosting time would be appreciated. >>=20 >> =3D Initial Committers =3D >>=20 >> * Alejandro Abdelnur (tucu at cloudera dot com) >> * Andre Arcilla (arcilla at yahoo-inc dot com) >> * Andrew Bayer (abayer at cloudera dot com) >> * Konstantin Boudnik (cos at apache dot org) >> * Eli Collins (eli at apache dot org) >> * Travis Crawford (travis at twitter dot com) >> * Bruno Mah=C3=A9 (bruno at cloudera dot com) >> * Alan Gates (gates at apache dot org) >> * Patrick Hunt (phunt at apache dot org) >> * Peter Linnell (plinnell at cloudera dot com) >> * Steve Loughran (stevel at apache dot org) >> * Owen O'Malley (omalley at apache dot org) >> * James Page (James.page at canonical dot com) >> * Roman Shaposhnik (rvs at cloudera dot com) >> * John Sichi (jvs at apache dot org) >> * Michael Stack (stack at apache dot org) >> * Tom White (tomwhite at apache dot org) >> * Andrei Savu (asavu at apache dot org) >> * Edward J. Yoon (edwardyoon at apache dot org) >>=20 >> =3D Affiliations =3D >>=20 >> * Alejandro Abdelnur, Cloudera >> * Andre Arcilla, Yahoo! Inc. >> * Andrew Bayer, Cloudera >> * Konstantin Boudnik, free lancer >> * Eli Collins, Cloudera >> * Travis Crawford, Twitter >> * Bruno Mah=C3=A9, Cloudera >> * Alan Gates, Yahoo! >> * Patrick Hunt, Cloudera >> * Peter Linnell, Cloudera >> * Steve Loughran, HP Laboratories >> * Owen O'Malley, Yahoo! >> * James Page, Canonical >> * Roman Shaposhnik, Cloudera >> * John Sichi, Facebook >> * Michael Stack, StumbleUpon >> * Tom White, Cloudera >> * Andrei Savu, Adobe >> * Edward J. Yoon, Korea Telecom >>=20 >> =3D Sponsors =3D >>=20 >> =3D=3D Champion =3D=3D >>=20 >> * Patrick Hunt >>=20 >> =3D=3D Nominated Mentors =3D=3D >>=20 >> * Patrick Hunt >> * Tom White >> * Owen O'Malley >> * Alan Gates >> * Steve Loughran >>=20 >> =3D=3D Sponsoring Entity =3D=3D >>=20 >> * Apache Incubator PMC >>=20 >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org >> For additional commands, e-mail: general-help@incubator.apache.org >>=20 --Apple-Mail-1--653335474--