Return-Path: X-Original-To: apmail-incubator-bigtop-commits-archive@minotaur.apache.org Delivered-To: apmail-incubator-bigtop-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4EE9C780A for ; Mon, 31 Oct 2011 00:30:51 +0000 (UTC) Received: (qmail 27633 invoked by uid 500); 31 Oct 2011 00:30:51 -0000 Delivered-To: apmail-incubator-bigtop-commits-archive@incubator.apache.org Received: (qmail 27597 invoked by uid 500); 31 Oct 2011 00:30:51 -0000 Mailing-List: contact bigtop-commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: bigtop-dev@incubator.apache.org Delivered-To: mailing list bigtop-commits@incubator.apache.org Received: (qmail 27570 invoked by uid 99); 31 Oct 2011 00:30:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Oct 2011 00:30:51 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Oct 2011 00:30:36 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 6D6A223888E4; Mon, 31 Oct 2011 00:30:13 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1195290 [1/6] - in /incubator/bigtop/branches/hadoop-0.22: ./ bigtop-deploy/ bigtop-deploy/puppet/ bigtop-deploy/puppet/manifests/ bigtop-deploy/puppet/modules/ bigtop-deploy/puppet/modules/hadoop-flume/ bigtop-deploy/puppet/modules/hadoop... Date: Mon, 31 Oct 2011 00:30:06 -0000 To: bigtop-commits@incubator.apache.org From: rvs@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20111031003013.6D6A223888E4@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: rvs Date: Mon Oct 31 00:30:01 2011 New Revision: 1195290 URL: http://svn.apache.org/viewvc?rev=1195290&view=rev Log: Merging the Bigtop 0.2.0 RC code Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/hadoop - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/hadoop.default incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/hadoop-env.sh incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/taskcontroller.cfg incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/lib/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/lib/facter/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/lib/facter/kadm_keytab.rb - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/manifests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/manifests/init.pp incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/templates/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/templates/kadm5.acl - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/templates/kdc.conf incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/templates/krb5.conf incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/tests/ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/tests/init.pp - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/hadoop.default incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/bigtop-utils/ incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/bigtop-utils/LICENSE incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/bigtop-utils/bigtop-detect-javahome - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/whirr/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/hadoop.nofiles.conf incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/hbase.nofiles.conf incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/zookeeper/zoo.cfg - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/ incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/compat incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/control - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/copyright incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/rules (contents, props changed) - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/zookeeper/rules incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/hadoop-sbin.postinst - copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/hadoop.daemon.postinst.tpl incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/ incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/RPMS/ incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/RPMS/.gitignore incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/SPECS/ incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/SPECS/.gitignore incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/SPECS/bigtop-utils.spec incubator/bigtop/branches/hadoop-0.22/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/UrpmiCmdLinePackageManager.groovy incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/package/src/main/resources/package_data_urpmi.xml Modified: incubator/bigtop/branches/hadoop-0.22/README incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/flume/install_flume.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/hadoop.default incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/install_hadoop.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/install_hbase.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/hadoop-hive.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/hadoop-hive.sh.suse incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/install_hive.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/mahout/install_mahout.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/oozie/create-package-layout incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/oozie/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/oozie/oozie.init incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/install_pig.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/sqoop/install_sqoop.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/sqoop/sqoop-metastore.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/sqoop/sqoop-metastore.sh.suse incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/install_whirr.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/zookeeper/hadoop-zookeeper.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/zookeeper/hadoop-zookeeper.sh.suse incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/zookeeper/install_zookeeper.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/flume/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/flume/service-init.d.tpl incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/hadoop-sbin.install incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/hadoop.install incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/rules incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/service-init.d.tpl incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hbase/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hbase/hadoop-hbase.install incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hbase/rules incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hbase/service-init.d.tpl incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hive/changelog incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hive/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/mahout/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/oozie/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/oozie/oozie.postinst incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/pig/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/pig/rules incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/whirr/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/whirr/rules incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/zookeeper/control incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/zookeeper/hadoop-zookeeper-server.init incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/zookeeper/rules incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/flume/SOURCES/init.d incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/flume/SOURCES/init.d.suse incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/flume/SPECS/flume.spec incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hadoop/SOURCES/hadoop-init.tmpl incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hadoop/SOURCES/hadoop-init.tmpl.suse incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hadoop/SPECS/hadoop.spec incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh.suse incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hive/SOURCES/install_hive.sh incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hive/SPECS/hive.spec incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/mahout/SPECS/mahout.spec incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/pig/SPECS/pig.spec incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/sqoop/SPECS/sqoop.spec incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/whirr/SPECS/whirr.spec incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/zookeeper/SPECS/zookeeper.spec incubator/bigtop/branches/hadoop-0.22/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy incubator/bigtop/branches/hadoop-0.22/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/RPMPackage.groovy incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/flume/pom.xml incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/hadoop/pom.xml incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/hbase/pom.xml incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/oozie/pom.xml incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/pom.xml incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-execution/smokes/hbase/pom.xml incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-execution/smokes/pom.xml incubator/bigtop/branches/hadoop-0.22/bigtop.mk incubator/bigtop/branches/hadoop-0.22/package.mk Modified: incubator/bigtop/branches/hadoop-0.22/README URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/README?rev=1195290&r1=1195289&r2=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/README (original) +++ incubator/bigtop/branches/hadoop-0.22/README Mon Oct 31 00:30:01 2011 @@ -49,6 +49,21 @@ On Debian-based systems one also needs * build-essential dh-make debhelper devscripts * reprepro +On openSUSE 11.4+, in addition, one also needs to ensure the following is installed: + + +relaxngDatatype +docbook-utils docbook-simple +asciidoc +fuse-devel +docbook5 +docbook5-xsl-stylesheets +libxml2-devel +xmlformat +xmlto +libxslt +libopenssl-devel + ## Building packages $ make [component-name]-[rpm|deb] Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp?rev=1195290&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp (added) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp Mon Oct 31 00:30:01 2011 @@ -0,0 +1,161 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +class hadoop_cluster_node { + $hadoop_namenode_host="$hadoop_head_node" + $hadoop_namenode_port="17020" + $hadoop_namenode_thrift_port="10090" + # $hadoop_dfs_namenode_plugins="org.apache.hadoop.thriftfs.NamenodePlugin" + # $hadoop_dfs_datanode_plugins="org.apache.hadoop.thriftfs.DatanodePlugin" + + $hadoop_jobtracker_host="$hadoop_head_node" + $hadoop_jobtracker_port="8021" + $hadoop_jobtracker_thrift_port="9290" + # $hadoop_mapred_jobtracker_plugins="org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin" + # $hadoop_mapred_tasktracker_plugins="org.apache.hadoop.mapred.TaskTrackerCmonInst" + + $hadoop_core_proxyusers = { oozie => { groups => 'root,hadoop,jenkins,oozie,users', hosts => "${hadoop_head_node},localhost,127.0.0.1" } } + + $hadoop_hbase_rootdir = "hdfs://$hadoop_namenode_host:$hadoop_namenode_port/hbase" + $hadoop_hbase_zookeeper_quorum = "$hadoop_head_node" + + $hadoop_zookeeper_ensemble = ["$hadoop_head_node:2888:3888"] + + $namenode_data_dirs = ["/mnt/namenode"] + $hdfs_data_dirs = ["/mnt/hdfs"] + $mapred_data_dirs = ["/mnt/scratch"] + + if ($hadoop_security_authentication == "kerberos") { + $kerberos_domain = "compute-1.internal" + $kerberos_realm = "EXAMPLE.COM" + $kerberos_kdc_server = "$hadoop_head_node" + + include kerberos::client + kerberos::client::host_keytab { ["hdfs", "mapred", "hbase", "oozie"]: + princs_map => { hdfs => [ "host", "hdfs" ], + mapred => [ "mapred" ], + hbase => [ "hbase" ], + oozie => [ "oozie" ], }, + } + } +} + +class hadoop_worker_node inherits hadoop_cluster_node { + hadoop::datanode { "datanode": + namenode_host => $hadoop_namenode_host, + namenode_port => $hadoop_namenode_port, + dirs => $hdfs_data_dirs, + auth => $hadoop_security_authentication, + } + + hadoop::tasktracker { "tasktracker": + namenode_host => $hadoop_namenode_host, + namenode_port => $hadoop_namenode_port, + jobtracker_host => $hadoop_jobtracker_host, + jobtracker_port => $hadoop_jobtracker_port, + dirs => $mapred_data_dirs, + auth => $hadoop_security_authentication, + } + + hadoop-hbase::server { "hbase region server": + rootdir => $hadoop_hbase_rootdir, + zookeeper_quorum => $hadoop_hbase_zookeeper_quorum, + kerberos_realm => $kerberos_realm, + } +} + +class hadoop_head_node inherits hadoop_cluster_node { + + if ($hadoop_security_authentication == "kerberos") { + include kerberos::kdc, kerberos::kdc::admin_server + } + + hadoop::namenode { "namenode": + port => $hadoop_namenode_port, + jobtracker_host => $hadoop_jobtracker_host, + jobtracker_port => $hadoop_jobtracker_port, + dirs => $namenode_data_dirs, + # thrift_port => $hadoop_namenode_thrift_port, + auth => $hadoop_security_authentication, + } + + hadoop::secondarynamenode { "secondary namenode": + namenode_host => $hadoop_namenode_host, + namenode_port => $hadoop_namenode_port, + auth => $hadoop_security_authentication, + } + + hadoop::jobtracker { "jobtracker": + namenode_host => $hadoop_namenode_host, + namenode_port => $hadoop_namenode_port, + host => $hadoop_jobtracker_host, + port => $hadoop_jobtracker_port, + dirs => $mapred_data_dirs, + # thrift_port => $hadoop_jobtracker_thrift_port, + auth => $hadoop_security_authentication, + } + + hadoop-hbase::master { "hbase master": + rootdir => $hadoop_hbase_rootdir, + zookeeper_quorum => $hadoop_hbase_zookeeper_quorum, + kerberos_realm => $kerberos_realm, + } + + hadoop-oozie::server { "oozie server": + kerberos_realm => $kerberos_realm, + } + + hadoop-zookeeper::server { "zookeeper": + myid => "0", + ensemble => $hadoop_zookeeper_ensemble, + } + + hadoop::create_hdfs_dirs { [ "/mapred", "/tmp", "/system", "/user", "/hbase", "/benchmarks", "/user/jenkins", "/user/hive" ]: + hdfs_dirs_meta => { "/tmp" => { perm => "777", user => "hdfs" }, + "/mapred" => { perm => "755", user => "mapred" }, + "/system" => { perm => "755", user => "hdfs" }, + "/user" => { perm => "755", user => "hdfs" }, + "/hbase" => { perm => "755", user => "hbase" }, + "/benchmarks" => { perm => "777", user => "hdfs" }, + "/user/jenkins" => { perm => "777", user => "jenkins"}, + "/user/hive" => { perm => "777", user => "hive" } }, + } +} + +class hadoop_gateway_node inherits hadoop_head_node { + # hadoop::client { "gateway": + # namenode_host => $hadoop_namenode_host, + # namenode_port => $hadoop_namenode_port, + # jobtracker_host => $hadoop_jobtracker_host, + # jobtracker_port => $hadoop_jobtracker_port, + # # auth => $hadoop_security_authentication, + # } + + hadoop-pig::client { "pig client": + } + hadoop-hive::client { "hive client": + hbase_zookeeper_quorum => $hadoop_hbase_zookeeper_quorum, + } + hadoop-sqoop::client { "sqoop client": + } + hadoop-oozie::client { "oozie client": + } + hadoop-hbase::client { "hbase client": + } + hadoop-zookeeper::client { "zookeeper client": + } + hadoop-flume::client { "flume client": + } +} Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,8 +13,28 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex +stage {"pre": before => Stage["main"]} -ant -Djavac.version=1.6 -Djava5.home=${JAVA5_HOME} -Dforrest.home=${FORREST_HOME} -Ddist.dir=debian/tmp -Dversion=${PIG_BASE_VERSION} package "$@" +yumrepo { "Bigtop": + baseurl => "http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-centos5/lastSuccessfulBuild/artifact/output/", + descr => "Bigtop packages", + enabled => 1, + gpgcheck => 0, +} +package { "jdk": + ensure => "installed", +} +# $hadoop_head_node="beefy.my.org" +# $hadoop_security_authentication="kerberos" + +import "cluster.pp" + +node default { + if ($hadoop_head_node == $fqdn) { + include hadoop_head_node + } else { + include hadoop_worker_node + } +} Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp Mon Oct 31 00:30:01 2011 @@ -12,24 +12,40 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -Source: sqoop -Section: misc -Priority: extra -Maintainer: Alex Newman -Build-Depends: debhelper (>= 6), ant, ant-optional, git-core, sed, asciidoc, xmlto -Standards-Version: 3.8.0 -Homepage: http://www.cloudera.com -Package: sqoop -Architecture: all -Depends: hadoop -Description: Tool for easy imports and exports of data sets between databases and HDFS - Sqoop is a tool that provides the ability to import and export data sets between - the Hadoop Distributed File System (HDFS) and relational databases. +class hadoop-flume { + define client { + package { "flume": + ensure => latest, + } + } -Package: sqoop-metastore -Architecture: all -Depends: sqoop (= ${source:Version}), adduser -Description: Shared metadata repository for Sqoop. - This optional package hosts a metadata server for Sqoop clients across a network to use. + # It really is flume node, but node is a reserved keyword for puppet + define agent { + package { "flume-node": + ensure => latest, + } + service { "flume-node": + ensure => running, + require => Package["flume-node"], + # FIXME: this need to be fixed in upstream flume + hasstatus => false, + hasrestart => true, + } + } + + define master { + package { "flume-master": + ensure => latest, + } + + service { "flume-master": + ensure => running, + require => Package["flume-node"], + # FIXME: this need to be fixed in upstream flume + hasstatus => true, + hasrestart => true, + } + } +} Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,7 +13,8 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex +hadoop-flume::client { "test-flume": } +hadoop-flume::agent { "test-flume-node": } +hadoop-flume::master { "test-flume-master": } -mvn clean source:jar install assembly:assembly -Pjavadoc site "$@" Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp?rev=1195290&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp (added) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp Mon Oct 31 00:30:01 2011 @@ -0,0 +1,71 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +class hadoop-hbase { + class client-package { + package { "hadoop-hbase": + ensure => latest, + } + } + + class common-server-config { + include client-package + + file { "/etc/hbase/conf/hbase-site.xml": + content => template("hadoop-hbase/hbase-site.xml"), + require => Package["hadoop-hbase"], + } + file { "/etc/hbase/conf/hbase-env.sh": + content => template("hadoop-hbase/hbase-env.sh"), + require => Package["hadoop-hbase"], + } + } + + define client { + include client-package + } + + define server($rootdir, $zookeeper_quorum, $kerberos_realm = "") { + include common-server-config + + package { "hadoop-hbase-regionserver": + ensure => latest, + } + + service { "hadoop-hbase-regionserver": + ensure => running, + require => Package["hadoop-hbase-regionserver"], + subscribe => File["/etc/hbase/conf/hbase-site.xml", "/etc/hbase/conf/hbase-env.sh"], + hasrestart => true, + hasstatus => true, + } + } + + define master($rootdir, $zookeeper_quorum, $kerberos_realm = "") { + include common-server-config + + package { "hadoop-hbase-master": + ensure => latest, + } + + service { "hadoop-hbase-master": + ensure => running, + require => Package["hadoop-hbase-master"], + subscribe => File["/etc/hbase/conf/hbase-site.xml", "/etc/hbase/conf/hbase-env.sh"], + hasrestart => true, + hasstatus => true, + } + } +} Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh?rev=1195290&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh (added) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh Mon Oct 31 00:30:01 2011 @@ -0,0 +1,76 @@ +# +#/** +# * Copyright 2007 The Apache Software Foundation +# * +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +# Set environment variables here. + +# The java implementation to use. Java 1.6 required. +# export JAVA_HOME=/usr/java/jdk1.6.0/ + +# Extra Java CLASSPATH elements. Optional. +export HBASE_CLASSPATH=/etc/hadoop/conf + +# The maximum amount of heap to use, in MB. Default is 1000. +# export HBASE_HEAPSIZE=1000 + +# Extra Java runtime options. +# Below are what we set by default. May only work with SUN JVM. +# For more on why as well as other possible settings, +# see http://wiki.apache.org/hadoop/PerformanceTuning +export HBASE_OPTS="$HBASE_OPTS -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode" + +# Uncomment below to enable java garbage collection logging. +# export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log" + +# Uncomment and adjust to enable JMX exporting +# See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access. +# More details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html +# +# export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" +# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101" +# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102" +# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103" +# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104" + +# File naming hosts on which HRegionServers will run. $HBASE_HOME/conf/regionservers by default. +# export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers + +# Extra ssh options. Empty by default. +# export HBASE_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR" + +# Where log files are stored. $HBASE_HOME/logs by default. +# export HBASE_LOG_DIR=${HBASE_HOME}/logs + +# A string representing this instance of hbase. $USER by default. +# export HBASE_IDENT_STRING=$USER + +# The scheduling priority for daemon processes. See 'man nice'. +# export HBASE_NICENESS=10 + +# The directory where pid files are stored. /tmp by default. +# export HBASE_PID_DIR=/var/hadoop/pids + +# Seconds to sleep between slave commands. Unset by default. This +# can be useful in large clusters, where, e.g., slave rsyncs can +# otherwise arrive faster than the master can service them. +# export HBASE_SLAVE_SLEEP=0.1 + +# Tell HBase whether it should manage it's own instance of Zookeeper or not. +# export HBASE_MANAGES_ZK=true Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml?rev=1195290&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml (added) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml Mon Oct 31 00:30:01 2011 @@ -0,0 +1,58 @@ + + + + + + hbase.cluster.distributed + true + + + + hbase.zookeeper.quorum + <%= zookeeper_quorum %> + + + + hbase.rootdir + <%= rootdir %> + + +<% if kerberos_realm != "" %> + + hbase.regionserver.kerberos.principal + hbase/_HOST@<%= kerberos_realm %> + + + hbase.regionserver.keytab.file + /etc/hbase.keytab + + + hbase.master.kerberos.principal + hbase/_HOST@<%= kerberos_realm %> + + + hbase.master.keytab.file + /etc/hbase.keytab + +<% end %> + Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,6 +13,14 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex - -mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@" +hadoop-hbase::client { "hbase shell": } +hadoop-hbase::master { "test-hbase-master": + rootdir => "hdfs://localhost:17020/hbase", + zookeeper_quorum => "localhost", + kerberos_realm => "KRB.YOU.ORG", +} +hadoop-hbase::server { "test-hbase-server": + rootdir => "hdfs://localhost:17020/hbase", + zookeeper_quorum => "localhost", + kerberos_realm => "KRB.YOU.ORG", +} Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,6 +13,15 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex +class hadoop-hive { + define client($hbase_master = "", $hbase_zookeeper_quorum = "") { + package { "hadoop-hive": + ensure => latest, + } -mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@" + file { "/etc/hive/conf/hive-site.xml": + content => template('hadoop-hive/hive-site.xml'), + require => Package["hadoop-hive"], + } + } +} Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml?rev=1195290&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml (added) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml Mon Oct 31 00:30:01 2011 @@ -0,0 +1,63 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +<% if hbase_master != "" %> + + hbase.master + <%= hbase_master %> + http://wiki.apache.org/hadoop/Hive/HBaseIntegration + +<% end %> + +<% if hbase_zookeeper_quorum != "" %> + + hbase.zookeeper.quorum + <%= hbase_zookeeper_quorum %> + http://wiki.apache.org/hadoop/Hive/HBaseIntegration + +<% end %> + + + javax.jdo.option.ConnectionURL + jdbc:derby:;databaseName=/var/lib/hive/metastore/metastore_db;create=true + JDBC connect string for a JDBC metastore + + + + javax.jdo.option.ConnectionDriverName + org.apache.derby.jdbc.EmbeddedDriver + Driver class name for a JDBC metastore + + + + hive.hwi.war.file + /usr/lib/hive/lib/hive-hwi-0.7.0-cdh3u1-SNAPSHOT.war + This is the WAR file with the jsp content for Hive Web Interface + + + Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,6 +13,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex - -ant -f src/build.xml package "$@" +hadoop-hive::client { "test-hive": } Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,8 +13,29 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex +class hadoop-oozie { + define client($kerberos_realm = "") { + package { "oozie-client": + ensure => latest, + } + } -ant -Djavac.version=1.6 -Djava5.home=${JAVA5_HOME} -Dforrest.home=${FORREST_HOME} -Ddist.dir=debian/tmp -Dversion=${PIG_BASE_VERSION} package "$@" + define server($kerberos_realm = "") { + package { "oozie": + ensure => latest, + } + file { "/etc/oozie/oozie-site.xml": + content => template("hadoop-oozie/oozie-site.xml"), + require => Package["oozie"], + } + service { "oozie": + ensure => running, + require => Package["oozie"], + hasrestart => true, + hasstatus => true, + } + + } +} Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml?rev=1195290&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml (added) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml Mon Oct 31 00:30:01 2011 @@ -0,0 +1,298 @@ + + + + + + + + + oozie.system.id + oozie-${user.name} + + The Oozie system ID. + + + + + oozie.systemmode + NORMAL + + System mode for Oozie at startup. + + + + + oozie.service.AuthorizationService.security.enabled + false + + Specifies whether security (user name/admin role) is enabled or not. + If disabled any user can manage Oozie system and manage any job. + + + + + oozie.service.PurgeService.older.than + 30 + + Jobs older than this value, in days, will be purged by the PurgeService. + + + + + oozie.service.PurgeService.purge.interval + 3600 + + Interval at which the purge service will run, in seconds. + + + + + oozie.service.CallableQueueService.queue.size + 10000 + Max callable queue size + + + + oozie.service.CallableQueueService.threads + 10 + Number of threads used for executing callables + + + + oozie.service.CallableQueueService.callable.concurrency + 3 + + Maximum concurrency for a given callable type. + Each command is a callable type (submit, start, run, signal, job, jobs, suspend,resume, etc). + Each action type is a callable type (Map-Reduce, Pig, SSH, FS, sub-workflow, etc). + All commands that use action executors (action-start, action-end, action-kill and action-check) use + the action type as the callable type. + + + + + oozie.service.coord.normal.default.timeout + + 120 + Default timeout for a coordinator action input check (in minutes) for normal job. + -1 means infinite timeout + + + + oozie.db.schema.name + oozie + + Oozie DataBase Name + + + + + oozie.service.StoreService.create.db.schema + true + + Creates Oozie DB. + + If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP. + If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up. + + + + + oozie.service.StoreService.jdbc.driver + org.apache.derby.jdbc.EmbeddedDriver + + JDBC driver class. + + + + + oozie.service.StoreService.jdbc.url + jdbc:derby:${oozie.data.dir}/${oozie.db.schema.name}-db;create=true + + JDBC URL. + + + + + oozie.service.StoreService.jdbc.username + sa + + DB user name. + + + + + oozie.service.StoreService.jdbc.password + + + DB user password. + + IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value, + if empty Configuration assumes it is NULL. + + IMPORTANT: if the StoreServicePasswordService is active, it will reset this value with the value given in + the console. + + + + + oozie.service.StoreService.pool.max.active.conn + 10 + + Max number of connections. + + + + + oozie.service.HadoopAccessorService.kerberos.enabled + <%= if (kerberos_realm != "") ; "true" else "false" end %> + + Indicates if Oozie is configured to use Kerberos. + + + + + local.realm + <%= kerberos_realm %> + + Kerberos Realm used by Oozie and Hadoop. Using 'local.realm' to be aligned with Hadoop configuration + + + + + oozie.service.HadoopAccessorService.keytab.file + /etc/oozie.keytab + + Location of the Oozie user keytab file. + + + + + oozie.service.HadoopAccessorService.kerberos.principal + ${user.name}/<%= fqdn %>@${local.realm} + + Kerberos principal for Oozie service. + + + + + oozie.service.HadoopAccessorService.jobTracker.whitelist + + + Whitelisted job tracker for Oozie service. + + + + + oozie.service.HadoopAccessorService.nameNode.whitelist + + + Whitelisted job tracker for Oozie service. + + + + + oozie.service.WorkflowAppService.system.libpath + /user/${user.name}/share/lib + + System library path to use for workflow applications. + This path is added to workflow application if their job properties sets + the property 'oozie.use.system.libpath' to true. + + + + + use.system.libpath.for.mapreduce.and.pig.jobs + false + + If set to true, submissions of MapReduce and Pig jobs will include + automatically the system library path, thus not requiring users to + specify where the Pig JAR files are. Instead, the ones from the system + library path are used. + + + + + oozie.authentication.type + simple + + Defines authentication used for Oozie HTTP endpoint. + Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME# + + + + + oozie.authentication.token.validity + 36000 + + Indicates how long (in seconds) an authentication token is valid before it has + to be renewed. + + + + + oozie.authentication.signature.secret + oozie + + The signature secret for signing the authentication tokens. + If not set a random secret is generated at startup time. + In order to authentiation to work correctly across multiple hosts + the secret must be the same across al the hosts. + + + + + oozie.authentication.cookie.domain + + + The domain to use for the HTTP cookie that stores the authentication token. + In order to authentiation to work correctly across multiple hosts + the domain must be correctly set. + + + + + oozie.authentication.simple.anonymous.allowed + true + + Indicates if anonymous requests are allowed. + This setting is meaningful only when using 'simple' authentication. + + + + + oozie.authentication.kerberos.principal + HTTP/<%= fqdn %>@${local.realm} + + Indicates the Kerberos principal to be used for HTTP endpoint. + The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification. + + + + + oozie.authentication.kerberos.keytab + ${oozie.service.HadoopAccessorService.keytab.file} + + Location of the keytab file with the credentials for the principal. + Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop. + + + + + Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,6 +13,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex - -ant -f src/build.xml package "$@" +hadoop-oozie::server { "test-oozie-server": } +hadoop-oozie::client { "test-oozie": } Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,6 +13,19 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex - -mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@" +class hadoop-pig { + define client { + package { "hadoop-pig": + ensure => latest, + require => Package["hadoop"], + } + + file { "/etc/pig/conf/pig.properties": + content => template('hadoop-pig/pig.properties'), + require => Package["hadoop-pig"], + owner => "root", /* FIXME: I'm really no sure about these -- we might end */ + mode => "755", /* up deploying/testing a different thing compared */ + /* to a straight rpm/deb deployment */ + } + } +} Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties?rev=1195290&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties (added) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties Mon Oct 31 00:30:01 2011 @@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Pig configuration file. All values can be overwritten by command line arguments. +# see bin/pig -help + +# log4jconf log4j configuration file +# log4jconf=./conf/log4j.properties + +# brief logging (no timestamps) +brief=false + +# clustername, name of the hadoop jobtracker. If no port is defined port 50020 will be used. +#cluster + +#debug level, INFO is default +debug=INFO + +# a file that contains pig script +#file= + +# load jarfile, colon separated +#jar= + +#verbose print all log messages to screen (default to print only INFO and above to screen) +verbose=false + +#exectype local|mapreduce, mapreduce is default +#exectype=mapreduce +# hod realted properties +#ssh.gateway +#hod.expect.root +#hod.expect.uselatest +#hod.command +#hod.config.dir +#hod.param + + +#Do not spill temp files smaller than this size (bytes) +pig.spill.size.threshold=5000000 +#EXPERIMENT: Activate garbage collection when spilling a file bigger than this size (bytes) +#This should help reduce the number of files being spilled. +pig.spill.gc.activation.size=40000000 + + +###################### +# Everything below this line is Yahoo specific. Note that I've made +# (almost) no changes to the lines above to make merging in from Apache +# easier. Any values I don't want from above I override below. +# +# This file is configured for use with HOD on the production clusters. If you +# want to run pig with a static cluster you will need to remove everything +# below this line and set the cluster value (above) to the +# hostname and port of your job tracker. + +exectype=mapreduce +log.file= Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,6 +13,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex - -ant -f src/build.xml package "$@" +include pig +pig::client { "test-pig": } Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,8 +13,30 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex +class hadoop-sqoop { + define client { + package { "sqoop": + ensure => latest, + } -ant -Djavac.version=1.6 -Djava5.home=${JAVA5_HOME} -Dforrest.home=${FORREST_HOME} -Ddist.dir=debian/tmp -Dversion=${PIG_BASE_VERSION} package "$@" + # FIXME: this is NOT supposed to be needed + # but for now that's the easiest way to add + # extra jars to sqoop classpath + file { "/usr/lib/sqoop/lib/mysql-connector-java-5.1.12.jar": + ensure => "/usr/share/java/mysql-connector-java-5.1.12.jar", + } + } + define metastore { + package { "sqoop-metastore": + ensure => latest, + } + service { "sqoop-metastore": + ensure => running, + require => Package["sqoop-metastore"], + hasstatus => true, + hasrestart => true, + } + } +} Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,6 +13,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex - -ant -f src/build.xml package "$@" +hadoop-sqoop::client { "test-sqoop": } +hadoop-sqoop::metastore { "test-sqoop-metastore": } Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp?rev=1195290&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp (added) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp Mon Oct 31 00:30:01 2011 @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +class hadoop-zookeeper { + define client { + package { "hadoop-zookeeper": + ensure => latest, + } + } + + define server($myid, $ensemble = ["localhost:2888:3888"]) { + package { "hadoop-zookeeper-server": + ensure => latest, + } + + service { "hadoop-zookeeper-server": + ensure => running, + require => Package["hadoop-zookeeper-server"], + subscribe => File["/etc/zookeeper/conf/zoo.cfg"], + hasrestart => true, + hasstatus => true, + } + + file { "/etc/zookeeper/conf/zoo.cfg": + content => template("hadoop-zookeeper/zoo.cfg"), + require => Package["hadoop-zookeeper-server"], + } + + file { "/var/lib/zookeeper/myid": + content => inline_template("<%= myid %>"), + require => Package["hadoop-zookeeper-server"], + } + } +} Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,6 +13,19 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex - -mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@" +maxClientCnxns=50 +# The number of milliseconds of each tick +tickTime=2000 +# The number of ticks that the initial +# synchronization phase can take +initLimit=10 +# The number of ticks that can pass between +# sending a request and getting an acknowledgement +syncLimit=5 +# the directory where the snapshot is stored. +dataDir=/var/lib/zookeeper +# the port at which the clients will connect +clientPort=2181 +<% ensemble.each_with_index do |server,idx| %> +server.<%= idx %>=<%= server %> +<% end %> Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp Mon Oct 31 00:30:01 2011 @@ -1,4 +1,3 @@ -#!/bin/sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,6 +13,8 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -ex - -mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@" +hadoop-zookeeper::client { "zoo visitor": } +hadoop-zookeeper::server { "test-oozie-server": + myid => "0", + ensemble => ["foo:2888:3888", "bar:2888:3888", "baz:2888:3888"], +} Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control&r1=1185814&r2=1195290&rev=1195290&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control (original) +++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb Mon Oct 31 00:30:01 2011 @@ -12,24 +12,29 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -Source: sqoop -Section: misc -Priority: extra -Maintainer: Alex Newman -Build-Depends: debhelper (>= 6), ant, ant-optional, git-core, sed, asciidoc, xmlto -Standards-Version: 3.8.0 -Homepage: http://www.cloudera.com - -Package: sqoop -Architecture: all -Depends: hadoop -Description: Tool for easy imports and exports of data sets between databases and HDFS - Sqoop is a tool that provides the ability to import and export data sets between - the Hadoop Distributed File System (HDFS) and relational databases. - -Package: sqoop-metastore -Architecture: all -Depends: sqoop (= ${source:Version}), adduser -Description: Shared metadata repository for Sqoop. - This optional package hosts a metadata server for Sqoop clients across a network to use. + +# We make the assumption hadoop's data files will be located in /data/ +# Puppet needs to know where they are +Facter.add("hadoop_storage_locations") do + setcode do + + data_dir_path = "/data/" + storage_locations = "" + + # We need to check /data/ exist + if File.directory?(data_dir_path) + + # We assume all data directory will be a number + Dir.foreach(data_dir_path) { |directory| + storage_locations += (data_dir_path + directory + ';') if directory =~ /\d+/ + } + end + + # Return the list of storage locations for hadoop + if storage_locations == "" + storage_locations = "/mnt" + end + storage_locations + end +end Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp?rev=1195290&view=auto ============================================================================== Files incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp (added) and incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp Mon Oct 31 00:30:01 2011 differ