Return-Path: X-Original-To: apmail-incubator-bigtop-commits-archive@minotaur.apache.org Delivered-To: apmail-incubator-bigtop-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C6C27721B for ; Tue, 1 Nov 2011 21:53:40 +0000 (UTC) Received: (qmail 43067 invoked by uid 500); 1 Nov 2011 21:53:40 -0000 Delivered-To: apmail-incubator-bigtop-commits-archive@incubator.apache.org Received: (qmail 42983 invoked by uid 500); 1 Nov 2011 21:53:40 -0000 Mailing-List: contact bigtop-commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: bigtop-dev@incubator.apache.org Delivered-To: mailing list bigtop-commits@incubator.apache.org Received: (qmail 42954 invoked by uid 99); 1 Nov 2011 21:53:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Nov 2011 21:53:40 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,LOTS_OF_MONEY X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Nov 2011 21:53:35 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id D1FCF2388A40; Tue, 1 Nov 2011 21:53:15 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1196308 [3/6] - in /incubator/bigtop/branches/hadoop-0.23: ./ bigtop-deploy/ bigtop-deploy/puppet/ bigtop-deploy/puppet/manifests/ bigtop-deploy/puppet/modules/ bigtop-deploy/puppet/modules/hadoop-flume/ bigtop-deploy/puppet/modules/hadoop... Date: Tue, 01 Nov 2011 21:53:09 -0000 To: bigtop-commits@incubator.apache.org From: rvs@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20111101215315.D1FCF2388A40@eris.apache.org> Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/sqoop/sqoop-metastore.sh.suse URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/sqoop/sqoop-metastore.sh.suse?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/sqoop/sqoop-metastore.sh.suse (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/sqoop/sqoop-metastore.sh.suse Tue Nov 1 21:53:04 2011 @@ -35,7 +35,14 @@ # Short-Description: Sqoop allows easy imports and exports of data sets between databases and the Hadoop Distributed File System (HDFS). ### END INIT INFO -source /lib/lsb/init-functions +. /lib/lsb/init-functions + +# Autodetect JAVA_HOME if not defined +if [ -e /usr/libexec/bigtop-detect-javahome ]; then + . /usr/libexec/bigtop-detect-javahome +elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then + . /usr/lib/bigtop-utils/bigtop-detect-javahome +fi STATUS_RUNNING=0 STATUS_DEAD=1 Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/whirr/do-component-build URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/whirr/do-component-build?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/whirr/do-component-build (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/whirr/do-component-build Tue Nov 1 21:53:04 2011 @@ -16,5 +16,7 @@ set -ex -mvn clean source:jar install assembly:assembly -Pjavadoc site "$@" +mvn clean -Pjavadoc site source:jar package assembly:assembly -DskipTests "$@" +mkdir build +tar -C build --strip-components=1 -xzf target/*.tar.gz Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/whirr/install_whirr.sh URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/whirr/install_whirr.sh?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/whirr/install_whirr.sh (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/whirr/install_whirr.sh Tue Nov 1 21:53:04 2011 @@ -1,4 +1,4 @@ -#!/bin/sh +#!/bin/bash # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with @@ -108,9 +108,21 @@ install -d -m 0755 $BIN_DIR cat > $BIN_DIR/whirr < $MAN_DIR/whirr.1.gz + +# Move the docs, but leave a symlink in place for compat. reasons +install -d -m 0755 $DOC_DIR +mv $LIB_DIR/docs/* $DOC_DIR +ln -s /${DOC_DIR/#$PREFIX/} $LIB_DIR/docs Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/zookeeper/hadoop-zookeeper.sh URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/zookeeper/hadoop-zookeeper.sh?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/zookeeper/hadoop-zookeeper.sh (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/common/zookeeper/hadoop-zookeeper.sh Tue Nov 1 21:53:04 2011 @@ -31,12 +31,20 @@ ### END INIT INFO set -e +# Autodetect JAVA_HOME if not defined +if [ -e /usr/libexec/bigtop-detect-javahome ]; then + . /usr/libexec/bigtop-detect-javahome +elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then + . /usr/lib/bigtop-utils/bigtop-detect-javahome +fi + PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin DAEMON_SCRIPT="/usr/bin/zookeeper-server" NAME=hadoop-zookeeper-server DESC="ZooKeeper daemon" -PID_FILE=/var/run/zookeeper/zookeeper-server.pid +# FIXME: a workaround for BIGTOP-207 +PID_FILE=/var/lib/zookeeper/zookeeper_server.pid install -d -m 0755 -o zookeeper -g zookeeper /var/run/zookeeper/ DODTIME=3 @@ -113,7 +121,8 @@ hadoop_stop_pidfile() { start() { - su -s /bin/sh zookeeper -c "${DAEMON_SCRIPT} start" + # FIXME: nohup is a workaround for BIGTOP-205 + su -s /bin/sh zookeeper -c "nohup ${DAEMON_SCRIPT} start >/dev/null 2>&1 > $PREFIX/$CONF_DIST_DIR/log4j.properties +# Define some default values that can be overridden by system properties +zookeeper.root.logger=INFO, CONSOLE +zookeeper.console.threshold=INFO +zookeeper.log.dir=. +zookeeper.log.file=zookeeper.log +zookeeper.log.threshold=DEBUG +zookeeper.tracelog.dir=. +zookeeper.tracelog.file=zookeeper_trace.log + +log4j.appender.ROLLINGFILE.File=\${zookeeper.log.dir}/\${zookeeper.log.file} +log4j.rootLogger=\${zookeeper.root.logger} +__EOT__ + # Copy in the /usr/bin/zookeeper-server wrapper install -d -m 0755 $PREFIX/$LIB_DIR/bin @@ -133,6 +148,14 @@ wrapper=$PREFIX/usr/bin/zookeeper-client install -d -m 0755 `dirname $wrapper` cat > $wrapper < $wrapper < +Build-Depends: debhelper (>= 7.0.0~) +Standards-Version: 3.8.0 +Homepage: http://incubator.apache.org/bigtop/ -set -ex - -ant -f src/build.xml package "$@" +Package: bigtop-utils +Architecture: all +Depends: ${misc:Depends} +Description: Collection of useful tools for Bigtop + This includes a collection of useful tools and files for Bigtop Added: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/bigtop-utils/copyright URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/bigtop-utils/copyright?rev=1196308&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/bigtop-utils/copyright (added) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/bigtop-utils/copyright Tue Nov 1 21:53:04 2011 @@ -0,0 +1,15 @@ +Format: http://dep.debian.net/deps/dep5 +Source: http://incubator.apache.org/bigtop/ +Upstream-Name: Apache Bigtop - Incubating + +Files: * +Copyright: 2008-2011, The Apache Software Foundation +License: Apache-2.0 + +Files debian/* +Copyright: 2011, The Apache Software Foundation +License: Apache-2.0 + +License: Apache-2.0 + On Debian systems, the complete text of the Apache 2.0 license + can be found in "/usr/share/common-licenses/Apache-2.0". Copied: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/bigtop-utils/rules (from r1196246, incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/rules) URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/bigtop-utils/rules?p2=incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/bigtop-utils/rules&p1=incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/rules&r1=1196246&r2=1196308&rev=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/rules (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/bigtop-utils/rules Tue Nov 1 21:53:04 2011 @@ -23,56 +23,52 @@ export DH_VERBOSE=1 # This has to be exported to make some magic below work. export DH_OPTIONS +DISTRO_CODENAME=$(shell lsb_release -sc) + patch: patch-stamp patch-stamp: touch $@ +build: + dh_testdir + dh_clean -k + + +build-indep: build-indep-stamp +build-indep-stamp: patch-stamp + touch $@ + clean: dh_testdir dh_testroot rm -f *-stamp dh_clean - rm -Rf debian/tmp debian/hadoop-zookeeper debian/zookeeper find debian -name .\*swp -exec rm -f {} \; -build-indep: build-indep-stamp -build-indep-stamp: patch-stamp - bash debian/do-component-build -Divy.home=`pwd`/debian/.ivy - mkdir -p debian/tmp - tar cf - --exclude=debian/\* . | (cd debian/tmp && tar xf -) - touch $@ - install: install-indep install-indep: - dh_testdir - dh_testroot - cp debian/zookeeper.1 . - sh -x debian/install_zookeeper.sh \ - --build-dir=. \ - --prefix=debian/hadoop-zookeeper - dh_install -i + dh_installdirs + # Copy repo file + install -d -p -m 755 debian/bigtop-utils/usr/lib/bigtop-utils/ + install -p -m 755 debian/bigtop-detect-javahome debian/bigtop-utils/usr/lib/bigtop-utils/ + dh_install + -binary-common: +binary-common: dh_testdir dh_testroot - dh_installchangelogs - dh_installdocs - dh_installman - dh_link - dh_strip dh_compress dh_fixperms dh_makeshlibs - dh_installinit + dh_installdocs dh_installdeb dh_shlibdeps dh_gencontrol dh_md5sums dh_builddeb -binary-indep: build-indep install-indep +binary-indep: build install install-indep $(MAKE) -f debian/rules DH_OPTIONS=-i binary-common - -binary-arch: binary: binary-indep -.PHONY: build clean binary-indep binary install-indep binary-arc + +.PHONY: clean install-indep binary-arc Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/flume/control URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/flume/control?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/flume/control (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/flume/control Tue Nov 1 21:53:04 2011 @@ -16,14 +16,14 @@ Source: flume Section: misc Priority: extra -Maintainer: Alex Newman +Maintainer: Bigtop Build-Depends: debhelper (>= 6), ant, ant-optional, git-core Standards-Version: 3.8.0 -Homepage: http://www.cloudera.com +Homepage: http://incubator.apache.org/projects/flume.html Package: flume Architecture: all -Depends: hadoop-zookeeper (>= 3.3.1), adduser, hadoop +Depends: hadoop-zookeeper (>= 3.3.1), adduser, hadoop, bigtop-utils Description: reliable, scalable, and manageable distributed data collection application Flume is a reliable, scalable, and manageable distributed data collection application for collecting data such as logs and delivering it to data stores Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/flume/service-init.d.tpl URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/flume/service-init.d.tpl?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/flume/service-init.d.tpl (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/flume/service-init.d.tpl Tue Nov 1 21:53:04 2011 @@ -27,6 +27,12 @@ # # description: Flume @FLUME_DAEMON@ +# Autodetect JAVA_HOME if not defined +if [ -e /usr/libexec/bigtop-detect-javahome ]; then + . /usr/libexec/bigtop-detect-javahome +elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then + . /usr/lib/bigtop-utils/bigtop-detect-javahome +fi export FLUME_HOME=/usr/lib/flume export FLUME_LOG_DIR=/var/log/flume Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/control URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/control?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/control (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/control Tue Nov 1 21:53:04 2011 @@ -15,14 +15,14 @@ Source: hadoop-hbase Section: misc Priority: extra -Maintainer: Alex Newman +Maintainer: Bigtop Build-Depends: debhelper (>= 6), git-core, adduser Standards-Version: 3.8.0 -Homepage: http://hadoop.apache.org/hbase/ +Homepage: http://hbase.apache.org/ Package: hadoop-hbase Architecture: all -Depends: adduser, hadoop-zookeeper (>= 3.3.1), hadoop +Depends: adduser, hadoop-zookeeper (>= 3.3.1), hadoop, bigtop-utils Recommends: ntp Description: HBase is the Hadoop database Use it when you need random, realtime read/write access to your Big Data. Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/hadoop-hbase.install URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/hadoop-hbase.install?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/hadoop-hbase.install (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/hadoop-hbase.install Tue Nov 1 21:53:04 2011 @@ -1,3 +1,4 @@ /usr/lib/hbase /usr/bin/hbase /etc/hbase +/etc/security/limits.d/hbase.nofiles.conf Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/rules URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/rules?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/rules (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/rules Tue Nov 1 21:53:04 2011 @@ -49,12 +49,14 @@ install: build --build-dir=target/hbase-0.92/hbase-0.92 \ --doc-dir=usr/share/doc/hadoop-hbase-doc \ --prefix=debian/tmp + mkdir -p debian/tmp/etc/security/limits.d + cp debian/hbase.nofiles.conf debian/tmp/etc/security/limits.d # Symlink in the dependency jars from their packages. Both of these packages # provide an unversioned symlink foo.jar -> foo-0.1.2.jar. rm debian/tmp/usr/lib/hbase/lib/zookeeper-*.jar rm debian/tmp/usr/lib/hbase/lib/hadoop-*.jar ln -s /usr/lib/zookeeper/zookeeper.jar debian/tmp/usr/lib/hbase/lib/zookeeper.jar - ln -s /usr/lib/hadoop/hadoop-core.jar debian/tmp/usr/lib/hbase/lib/hadoop-core.jar + ln -s /usr/lib/hadoop/hadoop-common.jar /usr/lib/hadoop/hadoop-hdfs.jar /usr/lib/hadoop/hadoop-mapred.jar debian/tmp/usr/lib/hbase/lib/ ln -s /var/log/hbase debian/tmp/usr/lib/hbase/logs ln -s /var/run/hbase debian/tmp/usr/lib/hbase/pids Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/service-init.d.tpl URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/service-init.d.tpl?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/service-init.d.tpl (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hbase/service-init.d.tpl Tue Nov 1 21:53:04 2011 @@ -42,7 +42,7 @@ set -e -source /etc/default/hadoop +. /etc/default/hadoop # Our default HBASE_HOME and HBASE_PID_DIR export HBASE_HOME=/usr/lib/hbase Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hive/changelog URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hive/changelog?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hive/changelog (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hive/changelog Tue Nov 1 21:53:04 2011 @@ -1,17 +1 @@ -hive (0.3.0-0cloudera0.3.0) cloudera; urgency=low - - * New upstream - - -- Todd Lipcon Thu, 04 Jun 2009 15:42:42 -0700 - -hive (0.3~svn759018-0cloudera0.3.0) cloudera; urgency=low - - * New upstream - - -- Mon, 06 Apr 2009 04:24:57 +0000 - -hive (0~svn748058-0cloudera0.3.0) cloudera; urgency=low - - * Initial release - - -- Todd Lipcon Thu, 02 Apr 2009 23:36:13 -0400 +-- NOTE: this gets auto-generated Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hive/control URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hive/control?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hive/control (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/hive/control Tue Nov 1 21:53:04 2011 @@ -16,14 +16,14 @@ Source: hadoop-hive Section: misc Priority: extra -Maintainer: Todd Lipcon +Maintainer: Bigtop Build-Depends: debhelper (>= 6), ant, python-support, git-core Standards-Version: 3.8.0 Homepage: http://hadoop.apache.org/hive/ Package: hadoop-hive Architecture: all -Depends: adduser, hadoop +Depends: adduser, hadoop, bigtop-utils Description: A data warehouse infrastructure built on top of Hadoop Hive is a data warehouse infrastructure built on top of Hadoop that provides tools to enable easy data summarization, adhoc querying and Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/mahout/control URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/mahout/control?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/mahout/control (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/mahout/control Tue Nov 1 21:53:04 2011 @@ -23,7 +23,7 @@ Homepage: http://mahout.apache.org Package: mahout Architecture: all -Depends: hadoop +Depends: hadoop, bigtop-utils Description: A set of Java libraries for scalable machine learning. Mahout's goal is to build scalable machine learning libraries. With scalable we mean: Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/oozie/control URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/oozie/control?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/oozie/control (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/oozie/control Tue Nov 1 21:53:04 2011 @@ -15,14 +15,15 @@ Source: oozie Section: misc Priority: extra -Maintainer: Arvind Prabhakar +Maintainer: Bigtop Build-Depends: debhelper (>= 6) Depends: zip, unzip Standards-Version: 3.8.0 -Homepage: http://archive.cloudera.com/cdh/3/oozie +Homepage: http://incubator.apache.org/oozie/ Package: oozie-client Architecture: all +Depends: bigtop-utils Description: Command line utility that allows remote access and operation of oozie. Using this utility, the user can deploy workflows and perform other administrative and @@ -31,7 +32,7 @@ Description: Command line utility that a Package: oozie Architecture: all -Depends: oozie-client (= ${source:Version}) +Depends: oozie-client (= ${source:Version}), hadoop, zip, unzip Description: A workflow and coordinator sytem for Hadoop jobs. Oozie workflows are actions arranged in a control dependency DAG (Direct Acyclic Graph). Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/oozie/oozie.postinst URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/oozie/oozie.postinst?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/oozie/oozie.postinst (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/oozie/oozie.postinst Tue Nov 1 21:53:04 2011 @@ -38,6 +38,7 @@ case "$1" in configure) chown -R oozie:oozie /var/lib/oozie /var/log/oozie update-rc.d oozie defaults >/dev/null || exit 1 + /usr/lib/oozie/bin/oozie-setup.sh -hadoop 0.20.200 /usr/lib/hadoop ;; abort-upgrade|abort-remove|abort-deconfigure) Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/pig/control URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/pig/control?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/pig/control (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/pig/control Tue Nov 1 21:53:04 2011 @@ -1,14 +1,14 @@ Source: hadoop-pig Section: misc Priority: extra -Maintainer: Todd Lipcon -Build-Depends: debhelper (>= 6), ant, ant-optional, git-core +Maintainer: Bigtop +Build-Depends: debhelper (>= 6), ant, ant-optional Standards-Version: 3.8.0 -Homepage: http://hadoop.apache.org/pig/ +Homepage: http://pig.apache.org/ Package: hadoop-pig Architecture: all -Depends: hadoop +Depends: hadoop, bigtop-utils Description: A platform for analyzing large data sets using Hadoop Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/pig/rules URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/pig/rules?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/pig/rules (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/pig/rules Tue Nov 1 21:53:04 2011 @@ -25,7 +25,6 @@ export DH_OPTIONS patch: patch-stamp patch-stamp: - patch -p1 < debian/patch touch $@ clean: @@ -50,7 +49,7 @@ install-indep: cp debian/*.properties . cp debian/pig.1 . sh -x debian/install_pig.sh \ - --build-dir=debian/tmp \ + --build-dir=build/pig-${PIG_BASE_VERSION} \ --prefix=debian/hadoop-pig \ --doc-dir=debian/hadoop-pig/usr/share/doc/hadoop-pig \ --examples-dir=debian/hadoop-pig/usr/share/doc/hadoop-pig/examples Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/sqoop/control URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/sqoop/control?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/sqoop/control (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/sqoop/control Tue Nov 1 21:53:04 2011 @@ -15,14 +15,14 @@ Source: sqoop Section: misc Priority: extra -Maintainer: Alex Newman +Maintainer: Bigtop Build-Depends: debhelper (>= 6), ant, ant-optional, git-core, sed, asciidoc, xmlto Standards-Version: 3.8.0 -Homepage: http://www.cloudera.com +Homepage: http://incubator.apache.org/sqoop/ Package: sqoop Architecture: all -Depends: hadoop +Depends: hadoop, bigtop-utils Description: Tool for easy imports and exports of data sets between databases and HDFS Sqoop is a tool that provides the ability to import and export data sets between the Hadoop Distributed File System (HDFS) and relational databases. Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/whirr/control URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/whirr/control?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/whirr/control (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/whirr/control Tue Nov 1 21:53:04 2011 @@ -15,13 +15,14 @@ Source: whirr Section: misc Priority: extra -Maintainer: Tom White +Maintainer: Bigtop Build-Depends: debhelper (>= 6), git-core Standards-Version: 3.8.0 -Homepage: http://incubator.apache.org/whirr +Homepage: http://whirr.apache.org/ Package: whirr Architecture: all +Depends: bigtop-utils Description: Scripts and libraries for running software services on cloud infrastructure Whirr provides . Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/whirr/rules URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/whirr/rules?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/whirr/rules (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/whirr/rules Tue Nov 1 21:53:04 2011 @@ -41,15 +41,15 @@ build-indep-stamp: patch-stamp mkdir -p debian/tmp # NOTE: FIXME me right away cp -r debian/whirr.1 . - tar cf - --exclude=debian/\* . | (cd debian/tmp && tar xf -) touch $@ install: install-indep install-indep: dh_testdir dh_testroot - sh -x debian/install_whirr.sh \ - --build-dir=debian/tmp \ + bash -x debian/install_whirr.sh \ + --build-dir=build \ + --doc-dir=debian/whirr/usr/share/doc/whirr \ --prefix=debian/whirr (dh_lintian) || /bin/true Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/control URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/control?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/control (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/control Tue Nov 1 21:53:04 2011 @@ -15,15 +15,16 @@ Source: hadoop-zookeeper Section: misc Priority: extra -Maintainer: Alex Newman -Build-Depends: debhelper (>= 6), ant, ant-optional, git-core, autoconf, automake, subversion +Maintainer: Bigtop +Build-Depends: debhelper (>= 6), ant, ant-optional, autoconf, automake Standards-Version: 3.8.0 -Homepage: http://hadoop.apache.org/zookeeper/ +Homepage: http://zookeeper.apache.org/ Package: hadoop-zookeeper Architecture: all Conflicts: zookeeper Replaces: zookeeper +Depends: bigtop-utils Description: A high-performance coordination service for distributed applications. ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them ,which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed. Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/hadoop-zookeeper-server.init URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/hadoop-zookeeper-server.init?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/hadoop-zookeeper-server.init (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/deb/zookeeper/hadoop-zookeeper-server.init Tue Nov 1 21:53:04 2011 @@ -36,7 +36,8 @@ DAEMON_SCRIPT="/usr/bin/zookeeper-server NAME=hadoop-zookeeper-server DESC="ZooKeeper daemon" -PID_FILE=/var/run/zookeeper/zookeeper-server.pid +# FIXME: a workaround for BIGTOP-207 +PID_FILE=/var/lib/zookeeper/zookeeper_server.pid install -d -m 0755 -o zookeeper -g zookeeper /var/run/zookeeper/ @@ -113,7 +114,8 @@ hadoop_stop_pidfile() { start() { - su -s /bin/sh zookeeper -c "${DAEMON_SCRIPT} start" + # FIXME: nohup is a workaround for BIGTOP-205 + su -s /bin/sh zookeeper -c "nohup ${DAEMON_SCRIPT} start >/dev/null 2>&1 = 3.3.1, hadoop >= 0.20.2 +Requires: coreutils, /usr/sbin/useradd, hadoop-zookeeper >= 3.3.1, hadoop >= 0.20.2 +Requires: bigtop-utils BuildRequires: ant xml-commons xml-commons-apis +%if 0%{?mgaversion} +Requires: bsh-utils +%else +Requires: sh-utils +%endif + %description Flume is a reliable, scalable, and manageable distributed data collection application for collecting data such as logs and delivering it to data stores such as Hadoop's HDFS. It can efficiently collect, aggregate, and move large amounts of log data. It has a simple, but flexible, architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. The system is centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic applications. @@ -73,19 +81,26 @@ Flume is a reliable, scalable, and manag Summary: The flume master daemon is the central administration and data path control point for flume nodes. Group: Development/Libraries BuildArch: noarch -Requires: %{name} = %{version}-%{release} -Requires: sh-utils, textutils, /usr/sbin/useradd, /sbin/chkconfig, /sbin/service +Requires: %{name} = %{version}-%{release}, /sbin/service Requires(post): /sbin/chkconfig -Requires(preun): /sbin/service, /sbin/chkconfig, %{name} +Requires(preun): /sbin/chkconfig %if %{?suse_version:1}0 # Required for init scripts Requires: insserv -%else +%endif + +%if 0%{?mgaversion} # Required for init scripts -Requires: redhat-lsb +Requires: initscripts %endif +# CentOS 5 does not have any dist macro +# So I will suppose anything that is not Mageia or a SUSE will be a RHEL/CentOS/Fedora +%if %{!?suse_version:1}0 && %{!?mgaversion:1}0 +# Required for init scripts +Requires: redhat-lsb +%endif %description master Flume is a reliable, scalable, and manageable distributed data collection application for collecting data such as logs and delivering it to data stores such as Hadoop's HDFS. It can efficiently collect, aggregate, and move large amounts of log data. It has a simple, but flexible, architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. The system is centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic applications. @@ -94,19 +109,26 @@ Flume is a reliable, scalable, and manag Summary: The flume node daemon is a core element of flume's data path and is responsible for generating, processing, and delivering data. Group: Development/Libraries BuildArch: noarch -Requires: %{name} = %{version}-%{release} -Requires: sh-utils, textutils, /usr/sbin/useradd, /sbin/chkconfig, /sbin/service +Requires: %{name} = %{version}-%{release}, /sbin/service Requires(post): /sbin/chkconfig -Requires(preun): /sbin/service, /sbin/chkconfig, %{name} +Requires(preun): /sbin/chkconfig %if %{?suse_version:1}0 # Required for init scripts Requires: insserv -%else +%endif + +%if 0%{?mgaversion} # Required for init scripts -Requires: redhat-lsb +Requires: initscripts %endif +# CentOS 5 does not have any dist macro +# So I will suppose anything that is not Mageia or a SUSE will be a RHEL/CentOS/Fedora +%if %{!?suse_version:1}0 && %{!?mgaversion:1}0 +# Required for init scripts +Requires: redhat-lsb +%endif %description node Flume is a reliable, scalable, and manageable distributed data collection application for collecting data such as logs and delivering it to data stores such as Hadoop's HDFS. It can efficiently collect, aggregate, and move large amounts of log data. It has a simple, but flexible, architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. The system is centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic applications. Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh Tue Nov 1 21:53:04 2011 @@ -31,8 +31,15 @@ ### END INIT INFO set -e -source /etc/default/hadoop -source /etc/default/hbase +. /etc/default/hadoop +. /etc/default/hbase + +# Autodetect JAVA_HOME if not defined +if [ -e /usr/libexec/bigtop-detect-javahome ]; then + . /usr/libexec/bigtop-detect-javahome +elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then + . /usr/lib/bigtop-utils/bigtop-detect-javahome +fi PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin DAEMON_SCRIPT="/usr/lib/hbase/bin/hbase-daemon.sh" Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh.suse URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh.suse?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh.suse (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh.suse Tue Nov 1 21:53:04 2011 @@ -32,10 +32,16 @@ # Short-Description: HBase @HBASE_DAEMON@ ### END INIT INFO -source /lib/lsb/init-functions -source /etc/default/hadoop -source /etc/default/hbase - +. /lib/lsb/init-functions +. /etc/default/hadoop +. /etc/default/hbase + +# Autodetect JAVA_HOME if not defined +if [ -e /usr/libexec/bigtop-detect-javahome ]; then + . /usr/libexec/bigtop-detect-javahome +elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then + . /usr/lib/bigtop-utils/bigtop-detect-javahome +fi STATUS_RUNNING=0 STATUS_DEAD=1 Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec Tue Nov 1 21:53:04 2011 @@ -20,7 +20,6 @@ %define logs_hbase %{hbase_home}/logs %define pids_hbase %{hbase_home}/pids %define webapps_hbase %{hbase_home}/hbase-webapps -%define doc_hbase %{_docdir}/hbase-%{hbase_version} %define man_dir %{_mandir} %define hbase_username hbase %define hbase_services master regionserver thrift @@ -42,11 +41,12 @@ /usr/lib/rpm/brp-compress ; \ %{nil} - +%define doc_hbase %{_docdir}/hbase %global initd_dir %{_sysconfdir}/rc.d %else +%define doc_hbase %{_docdir}/hbase-%{hbase_version} %global initd_dir %{_sysconfdir}/rc.d/init.d %endif @@ -66,8 +66,17 @@ Source2: install_hbase.sh Source3: hadoop-hbase.sh Source4: hadoop-hbase.sh.suse Source5: hbase.default +Source6: hbase.nofiles.conf BuildArch: noarch -Requires: sh-utils, textutils, /usr/sbin/useradd, /sbin/chkconfig, /sbin/service, hadoop-zookeeper, hadoop >= 0.20.2, hadoop-zookeeper >= 3.3.1 +Requires: coreutils, /usr/sbin/useradd, /sbin/chkconfig, /sbin/service +Requires: hadoop >= 0.20.2, hadoop-zookeeper >= 3.3.1, bigtop-utils + +%if 0%{?mgaversion} +Requires: bsh-utils +%else +Requires: sh-utils +%endif + %description HBase is an open-source, distributed, column-oriented store modeled after Google' Bigtable: A Distributed Storage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Hadoop. HBase includes: @@ -90,11 +99,19 @@ Requires: %{name} = %{version}-%{release %if %{?suse_version:1}0 # Required for init scripts Requires: insserv -%else +%endif + +%if 0%{?mgaversion} # Required for init scripts -Requires: redhat-lsb +Requires: initscripts %endif +# CentOS 5 does not have any dist macro +# So I will suppose anything that is not Mageia or a SUSE will be a RHEL/CentOS/Fedora +%if %{!?suse_version:1}0 && %{!?mgaversion:1}0 +# Required for init scripts +Requires: redhat-lsb +%endif %description master HMaster is the "master server" for a HBase. There is only one HMaster for a single HBase deployment. @@ -108,7 +125,16 @@ Requires: %{name} = %{version}-%{release %if %{?suse_version:1}0 # Required for init scripts Requires: insserv -%else +%endif + +%if 0%{?mgaversion} +# Required for init scripts +Requires: initscripts +%endif + +# CentOS 5 does not have any dist macro +# So I will suppose anything that is not Mageia or a SUSE will be a RHEL/CentOS/Fedora +%if %{!?suse_version:1}0 && %{!?mgaversion:1}0 # Required for init scripts Requires: redhat-lsb %endif @@ -126,7 +152,16 @@ Requires: %{name} = %{version}-%{release %if %{?suse_version:1}0 # Required for init scripts Requires: insserv -%else +%endif + +%if 0%{?mgaversion} +# Required for init scripts +Requires: initscripts +%endif + +# CentOS 5 does not have any dist macro +# So I will suppose anything that is not Mageia or a SUSE will be a RHEL/CentOS/Fedora +%if %{!?suse_version:1}0 && %{!?mgaversion:1}0 # Required for init scripts Requires: redhat-lsb %endif @@ -164,6 +199,9 @@ sh %{SOURCE2} \ %__install -d -m 0755 $RPM_BUILD_ROOT/etc/default/ %__install -m 0644 $RPM_SOURCE_DIR/hbase.default $RPM_BUILD_ROOT/etc/default/hbase +%__install -d -m 0755 $RPM_BUILD_ROOT/etc/security/limits.d +%__install -m 0644 %{SOURCE6} $RPM_BUILD_ROOT/etc/security/limits.d/hbase.nofiles.conf + %__install -d -m 0755 %{buildroot}/%{_localstatedir}/log/hbase ln -s %{_localstatedir}/log/hbase %{buildroot}/%{logs_hbase} @@ -189,7 +227,7 @@ done # Pull zookeeper and hadoop from their packages rm -f $RPM_BUILD_ROOT/usr/lib/hbase/lib/hadoop* \ $RPM_BUILD_ROOT/usr/lib/hbase/lib/zookeeper* -ln -s /usr/lib/hadoop/hadoop-core.jar $RPM_BUILD_ROOT/usr/lib/hbase/lib/hadoop-core.jar +ln -s /usr/lib/hadoop/hadoop-common.jar /usr/lib/hadoop/hadoop-hdfs.jar /usr/lib/hadoop/hadoop-mapred.jar $RPM_BUILD_ROOT/usr/lib/hbase/lib ln -s /usr/lib/zookeeper/zookeeper.jar $RPM_BUILD_ROOT/usr/lib/hbase/lib/zookeeper.jar %pre @@ -208,6 +246,7 @@ getent passwd hbase 2>&1 > /dev/null || %defattr(-,root,root) %config(noreplace) %{_sysconfdir}/default/hbase +%config(noreplace) /etc/security/limits.d/hbase.nofiles.conf %{hbase_home} %{hbase_home}/hbase-*.jar %{webapps_hbase} Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hive/SOURCES/install_hive.sh URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hive/SOURCES/install_hive.sh?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hive/SOURCES/install_hive.sh (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/hive/SOURCES/install_hive.sh Tue Nov 1 21:53:04 2011 @@ -120,7 +120,15 @@ do cat >>$wrapper <= 0.20.1 +Requires: hadoop >= 0.20.2, bigtop-utils Obsoletes: %{name}-webinterface %description Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/mahout/SPECS/mahout.spec URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/mahout/SPECS/mahout.spec?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/mahout/SPECS/mahout.spec (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/mahout/SPECS/mahout.spec Tue Nov 1 21:53:04 2011 @@ -20,11 +20,12 @@ %define log_mahout /var/log/%{mahout_name} %define bin_mahout /usr/bin %define man_dir /usr/share/man -%define doc_mahout %{_docdir}/mahout-%{mahout_version} %if %{?suse_version:1}0 +%define doc_mahout %{_docdir}/mahout %define alternatives_cmd update-alternatives %else +%define doc_mahout %{_docdir}/mahout-%{mahout_version} %define alternatives_cmd alternatives %endif @@ -44,7 +45,7 @@ Source0: %{name}-distribution-%{mahout_b Source1: do-component-build Source2: install_%{name}.sh Patch0: patch -Requires: hadoop >= 0.20.2, /sbin/chkconfig +Requires: hadoop >= 0.20.2, bigtop-utils %description Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec Tue Nov 1 21:53:04 2011 @@ -19,8 +19,10 @@ %define data_oozie /var/lib/oozie %if %{!?suse_version:1}0 + %define doc_oozie %{_docdir}/oozie-%{oozie_version} %define initd_dir %{_sysconfdir}/rc.d/init.d %else + %define doc_oozie %{_docdir}/oozie %define initd_dir %{_sysconfdir}/rc.d %endif @@ -28,7 +30,7 @@ Name: oozie Version: %{oozie_version} Release: %{oozie_release} Summary: Oozie is a system that runs workflows of Hadoop jobs. -URL: http://www.cloudera.com +URL: http://incubator.apache.org/oozie/ Group: Development/Libraries Buildroot: %{_topdir}/INSTALL/%{name}-%{version} License: APL2 @@ -37,9 +39,9 @@ Source1: do-component-build Source2: create-package-layout Patch0: patch Requires(pre): /usr/sbin/groupadd, /usr/sbin/useradd -Requires(post): /sbin/chkconfig +Requires(post): /sbin/chkconfig, hadoop Requires(preun): /sbin/chkconfig, /sbin/service -Requires: oozie-client = %{version} +Requires: zip, unzip, oozie-client = %{version} BuildArch: noarch %description @@ -85,10 +87,12 @@ BuildArch: noarch Version: %{version} Release: %{release} Summary: Client for Oozie Workflow Engine -URL: http://www.cloudera.com +URL: http://incubator.apache.org/oozie/ Group: Development/Libraries License: APL2 BuildArch: noarch +Requires: bigtop-utils + %description client Oozie client is a command line client utility that allows remote @@ -113,7 +117,7 @@ BuildArch: noarch %install %__rm -rf $RPM_BUILD_ROOT - sh %{SOURCE2} --extra-dir=$RPM_SOURCE_DIR --build-dir=. --server-dir=$RPM_BUILD_ROOT --client-dir=$RPM_BUILD_ROOT --docs-dir=$RPM_BUILD_ROOT%{_docdir}/%{name}-%{version} --initd-dir=$RPM_BUILD_ROOT%{initd_dir} + sh %{SOURCE2} --extra-dir=$RPM_SOURCE_DIR --build-dir=. --server-dir=$RPM_BUILD_ROOT --client-dir=$RPM_BUILD_ROOT --docs-dir=$RPM_BUILD_ROOT%{doc_oozie} --initd-dir=$RPM_BUILD_ROOT%{initd_dir} %__install -d -m 0755 $RPM_BUILD_ROOT/usr/bin @@ -125,6 +129,7 @@ getent group oozie >/dev/null || /usr/sb getent passwd oozie >/dev/null || /usr/sbin/useradd --comment "Oozie User" --shell /bin/false -M -r -g oozie --home /var/run/oozie oozie >/dev/null %post +%{lib_oozie}/bin/oozie-setup.sh -hadoop 0.20.200 /usr/lib/hadoop /sbin/chkconfig --add oozie %preun @@ -147,6 +152,7 @@ fi %{lib_oozie}/bin/oozie-stop.sh %{lib_oozie}/bin/oozie-sys.sh %{lib_oozie}/bin/oozie-env.sh +%{lib_oozie}/bin/oozied.sh %{lib_oozie}/oozie.war %{lib_oozie}/oozie-sharelib.tar.gz %{lib_oozie}/oozie-server @@ -164,7 +170,5 @@ fi %{lib_oozie}/bin/oozie %{lib_oozie}/bin/oozie-examples.sh %{lib_oozie}/lib -%{_docdir} -%docdir %{_docdir} +%doc %{doc_oozie} %{man_dir}/man1/oozie.1.* - Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/pig/SPECS/pig.spec URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/pig/SPECS/pig.spec?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/pig/SPECS/pig.spec (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/pig/SPECS/pig.spec Tue Nov 1 21:53:04 2011 @@ -19,7 +19,6 @@ %define log_pig /var/log/%{pig_name} %define bin_pig /usr/bin %define pig_config_virtual pig_active_configuration -%define doc_pig %{_docdir}/pig-%{pig_version} %define man_dir %{_mandir} # CentOS 5 does not have any dist macro @@ -38,6 +37,7 @@ /usr/lib/rpm/brp-python-bytecompile ; \ %{nil} +%define doc_pig %{_docdir}/pig-%{pig_version} %define alternatives_cmd alternatives %endif @@ -50,6 +50,7 @@ %define suse_check \# Define an empty suse_check for compatibility with older sles %endif +%define doc_pig %{_docdir}/pig %define alternatives_cmd update-alternatives %define __os_install_post \ %{suse_check} ; \ @@ -60,6 +61,7 @@ %if 0%{?mgaversion} +%define doc_pig %{_docdir}/pig-%{pig_version} %define alternatives_cmd update-alternatives %endif @@ -72,7 +74,6 @@ License: Apache License v2.0 URL: http://hadoop.apache.org/pig/ Group: Development/Libraries Buildroot: %{_topdir}/INSTALL/%{name}-%{version} -BuildRequires: /usr/bin/git BuildArch: noarch Source0: pig-%{pig_base_version}.tar.gz Source1: do-component-build @@ -80,8 +81,7 @@ Source2: install_pig.sh Source3: log4j.properties Source4: pig.1 Source5: pig.properties -Patch0: patch -Requires: hadoop +Requires: hadoop, bigtop-utils %description Pig is a platform for analyzing large data sets that consists of a high-level language @@ -108,7 +108,6 @@ language called Pig Latin, which has the %prep %setup -n apache-pig-e42a1e6 -%patch0 -p1 %build env PIG_BASE_VERSION=%{pig_base_version} bash %{SOURCE1} @@ -124,7 +123,7 @@ cp $RPM_SOURCE_DIR/log4j.properties . cp $RPM_SOURCE_DIR/pig.1 . cp $RPM_SOURCE_DIR/pig.properties . sh -x %{SOURCE2} \ - --build-dir=. \ + --build-dir=build/pig-%{pig_base_version} \ --doc-dir=$RPM_BUILD_ROOT%{doc_pig} \ --prefix=$RPM_BUILD_ROOT Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/sqoop/SPECS/sqoop.spec URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/sqoop/SPECS/sqoop.spec?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/sqoop/SPECS/sqoop.spec (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/sqoop/SPECS/sqoop.spec Tue Nov 1 21:53:04 2011 @@ -12,16 +12,17 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -%define doc_sqoop %{_docdir}/sqoop-%{sqoop_version} %define lib_sqoop /usr/lib/sqoop %if %{?suse_version:1}0 +%define doc_sqoop %{_docdir}/sqoop %global initd_dir %{_sysconfdir}/rc.d %else +%define doc_sqoop %{_docdir}/sqoop-%{sqoop_version} %global initd_dir %{_sysconfdir}/rc.d/init.d %endif @@ -31,7 +32,7 @@ Name: sqoop Version: %{sqoop_version} Release: %{sqoop_release} Summary: Sqoop allows easy imports and exports of data sets between databases and the Hadoop Distributed File System (HDFS). -URL: http://www.cloudera.com +URL: http://incubator.apache.org/sqoop/ Group: Development/Libraries Buildroot: %{_topdir}/INSTALL/%{name}-%{version} License: APL2 @@ -42,15 +43,15 @@ Source3: sqoop-metastore.sh Source4: sqoop-metastore.sh.suse Patch0: patch Buildarch: noarch -BuildRequires: asciidoc, xmlto -Prereq: hadoop +BuildRequires: asciidoc, xmlto +Requires: hadoop, bigtop-utils %description Sqoop allows easy imports and exports of data sets between databases and the Hadoop Distributed File System (HDFS). %package metastore Summary: Shared metadata repository for Sqoop. -URL: http://www.cloudera.com +URL: http://incubator.apache.org/sqoop/ Group: System/Daemons Provides: sqoop-metastore Requires: sqoop = %{version}-%{release} @@ -58,7 +59,16 @@ Requires: sqoop = %{version}-%{release} %if %{?suse_version:1}0 # Required for init scripts Requires: insserv -%else +%endif + +%if 0%{?mgaversion} +# Required for init scripts +Requires: initscripts +%endif + +# CentOS 5 does not have any dist macro +# So I will suppose anything that is not Mageia or a SUSE will be a RHEL/CentOS/Fedora +%if %{!?suse_version:1}0 && %{!?mgaversion:1}0 # Required for init scripts Requires: redhat-lsb %endif Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/whirr/SPECS/whirr.spec URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/whirr/SPECS/whirr.spec?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/whirr/SPECS/whirr.spec (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/whirr/SPECS/whirr.spec Tue Nov 1 21:53:04 2011 @@ -15,6 +15,12 @@ %define lib_whirr /usr/lib/whirr %define man_dir /usr/share/man +%if %{?suse_version:1}0 + %define doc_whirr %{_docdir}/whirr +%else + %define doc_whirr %{_docdir}/whirr-%{whirr_version} +%endif + # disable repacking jars %define __os_install_post %{nil} @@ -31,6 +37,7 @@ Source0: %{name}-%{whirr_base_version}-i Source1: do-component-build Source2: install_%{name}.sh Source3: whirr.1 +Requires: bigtop-utils %description Whirr provides @@ -52,13 +59,14 @@ bash %{SOURCE1} %install %__rm -rf $RPM_BUILD_ROOT cp $RPM_SOURCE_DIR/whirr.1 . -sh %{SOURCE2} \ - --build-dir=. \ +bash %{SOURCE2} \ + --build-dir=build \ + --doc-dir=$RPM_BUILD_ROOT%{doc_whirr} \ --prefix=$RPM_BUILD_ROOT %files %defattr(-,root,root) %attr(0755,root,root) %{lib_whirr} %attr(0755,root,root) %{_bindir}/%{name} -%doc %attr(0644,root,root) %{man_dir}/man1/whirr.1.gz -%ghost %{lib_whirr}/cli/whirr.log +%doc %attr(0644,root,root) %{man_dir}/man1/whirr.1.* +%doc %{doc_whirr} Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/zookeeper/SPECS/zookeeper.spec URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/zookeeper/SPECS/zookeeper.spec?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/zookeeper/SPECS/zookeeper.spec (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-packages/src/rpm/zookeeper/SPECS/zookeeper.spec Tue Nov 1 21:53:04 2011 @@ -14,7 +14,6 @@ # limitations under the License. %define etc_zookeeper /etc/zookeeper %define bin_zookeeper %{_bindir} -%define doc_zookeeper %{_docdir}/zookeeper-%{zookeeper_version} %define lib_zookeeper /usr/lib/zookeeper %define log_zookeeper /var/log/zookeeper %define run_zookeeper /var/run/zookeeper @@ -38,6 +37,7 @@ %{nil} +%define doc_zookeeper %{_docdir}/zookeeper %define alternatives_cmd update-alternatives %define alternatives_dep update-alternatives %define chkconfig_dep aaa_base @@ -46,6 +46,7 @@ %else +%define doc_zookeeper %{_docdir}/zookeeper-%{zookeeper_version} %define alternatives_cmd alternatives %define alternatives_dep chkconfig %define chkconfig_dep chkconfig @@ -70,11 +71,13 @@ Source2: install_zookeeper.sh Source3: hadoop-zookeeper.sh Source4: hadoop-zookeeper.sh.suse Source5: zookeeper.1 +Source6: zoo.cfg BuildArch: noarch -BuildRequires: ant, autoconf, automake, subversion -Requires(pre): coreutils, shadow-utils +BuildRequires: ant, autoconf, automake +Requires(pre): coreutils, shadow-utils, /usr/sbin/groupadd, /usr/sbin/useradd Requires(post): %{alternatives_dep} Requires(preun): %{alternatives_dep} +Requires: bigtop-utils %description ZooKeeper is a centralized service for maintaining configuration information, @@ -98,11 +101,21 @@ BuildArch: noarch %if %{?suse_version:1}0 # Required for init scripts Requires: insserv -%else +%endif + +%if 0%{?mgaversion} +# Required for init scripts +Requires: initscripts +%endif + +# CentOS 5 does not have any dist macro +# So I will suppose anything that is not Mageia or a SUSE will be a RHEL/CentOS/Fedora +%if %{!?suse_version:1}0 && %{!?mgaversion:1}0 # Required for init scripts Requires: redhat-lsb %endif + %description server This package starts the zookeeper server on startup @@ -114,7 +127,7 @@ bash %{SOURCE1} -Dversion=%{version} %install %__rm -rf $RPM_BUILD_ROOT -cp $RPM_SOURCE_DIR/zookeeper.1 . +cp $RPM_SOURCE_DIR/zookeeper.1 $RPM_SOURCE_DIR/zoo.cfg . sh %{SOURCE2} \ --build-dir=. \ --doc-dir=%{doc_zookeeper} \ Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy Tue Nov 1 21:53:04 2011 @@ -213,6 +213,8 @@ public abstract class PackageManager { return new YumCmdLinePackageManager(); case ~/(?is).*(suse|sles|sled).*/: return new ZypperCmdLinePackageManager(); + case ~/(?is).*(mageia).*/: + return new UrpmiCmdLinePackageManager(); default: return null; } Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/RPMPackage.groovy URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/RPMPackage.groovy?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/RPMPackage.groovy (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/RPMPackage.groovy Tue Nov 1 21:53:04 2011 @@ -44,7 +44,7 @@ class RPMPackage extends ManagedPackage if (curMetaKey == "description" || ((it =~ /^\s+: /).find() && curMetaKey == "summary")) { pkg.meta[curMetaKey] <<= "\n${it.replaceAll(/^\s+:/,'')}"; } else { - def m = (it =~ /(\S+) *: *(.+)/); + def m = (it =~ /(\S+)\s*:\s*(.*)/); if (m.size()) { String[] matcher = m[0]; if ("Name" == matcher[1] && !givenPkg) { Added: incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/UrpmiCmdLinePackageManager.groovy URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/UrpmiCmdLinePackageManager.groovy?rev=1196308&view=auto ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/UrpmiCmdLinePackageManager.groovy (added) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/UrpmiCmdLinePackageManager.groovy Tue Nov 1 21:53:04 2011 @@ -0,0 +1,76 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.bigtop.itest.pmanager + +import org.apache.bigtop.itest.posix.Service + +class UrpmiCmdLinePackageManager extends PackageManager { + static String type = "urpmi"; + + public void setDefaults(String defaults) {} + + public int addBinRepo(String record, String url, String key, String cookie) { + shRoot.exec("urpmi.addmedia '${record}' $url"); + return shRoot.getRet(); + } + + public int removeBinRepo(String record) { + shRoot.exec("urpmi.removemedia -q '${record}'"); + return shRoot.getRet(); + } + + public int refresh() { + shRoot.exec("urpmi.update -a"); + return shRoot.getRet(); + } + + public int cleanup() { + shRoot.exec("urpmi --clean"); + return shRoot.getRet(); + } + + public List search(String name) { + def packages = new ArrayList(); + shUser.exec("urpmq ${name} | sed -e 's/|/\\n/g' | uniq").out.each { + packages.add(PackageInstance.getPackageInstance (this, it)) + } + return packages + } + + public List lookup(String name) { + shUser.exec("urpmq -i $name | sed -re \"s/(Size\\s+:\\s[0-9]+)\\s+/\\1\\n/\" "); + return (shUser.getRet() == 0) ? RPMPackage.parseMetaOutput(null, shUser.out, this) : []; + } + + public int install(PackageInstance pkg) { + shRoot.exec("urpmi --verbose --force --auto ${pkg.name}"); + pkg.installMessages = shRoot.getOut().join('\n'); + return shRoot.getRet(); + } + + public int remove(PackageInstance pkg) { + shRoot.exec("urpme --verbose --force --auto ${pkg.name}"); + return shRoot.getRet(); + } + + public boolean isInstalled(PackageInstance pkg) { + shUser.exec("rpm -q ${pkg.name}") + return (shUser.getRet() == 0) + } +} Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/flume/pom.xml URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/flume/pom.xml?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/flume/pom.xml (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/flume/pom.xml Tue Nov 1 21:53:04 2011 @@ -33,8 +33,7 @@ org.apache.hadoop - hadoop-core - 0.20.2 + hadoop-common Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/pom.xml URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/pom.xml?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/pom.xml (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/pom.xml Tue Nov 1 21:53:04 2011 @@ -33,13 +33,15 @@ org.apache.hadoop - hadoop-core - 0.20.2 + hadoop-common org.apache.hadoop - hadoop-test - 0.20.2 + hadoop-common-test + + + org.apache.hadoop + hadoop-hdfs-test Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy Tue Nov 1 21:53:04 2011 @@ -18,7 +18,7 @@ package org.apache.bigtop.itest.hadoopexamples -import org.junit.Before +import org.junit.BeforeClass import static org.junit.Assert.assertNotNull import org.apache.bigtop.itest.shell.Shell import static org.junit.Assert.assertTrue @@ -28,6 +28,11 @@ import org.apache.bigtop.itest.JarConten import org.apache.commons.logging.LogFactory import org.apache.commons.logging.Log +import org.apache.bigtop.itest.junit.OrderedParameterized +import org.junit.runners.Parameterized.Parameters +import org.junit.runner.RunWith + +@RunWith(OrderedParameterized.class) class TestHadoopExamples { static private Log LOG = LogFactory.getLog(TestHadoopExamples.class); @@ -35,15 +40,20 @@ class TestHadoopExamples { private static final String HADOOP_CONF_DIR = System.getenv('HADOOP_CONF_DIR'); private static String hadoopExamplesJar = JarContent.getJarName(HADOOP_HOME, 'hadoop.*examples.*.jar'); + private static String hadoopMapredTestJar = + JarContent.getJarName(HADOOP_HOME, 'hadoop.*mapred-test.*.jar'); static { assertNotNull("HADOOP_HOME has to be set to run this test", HADOOP_HOME); assertNotNull("HADOOP_CONF_DIR has to be set to run this test", HADOOP_CONF_DIR); assertNotNull("Can't find hadoop-examples.jar file", hadoopExamplesJar); + assertNotNull("Can't find hadoop-mapred-test.jar file", hadoopMapredTestJar); } static final String HADOOP_EXAMPLES_JAR = HADOOP_HOME + "/" + hadoopExamplesJar; + static final String HADOOP_MR_TEST_JAR = + HADOOP_HOME + "/" + hadoopMapredTestJar; private static final String hadoop = "$HADOOP_HOME/bin/hadoop"; static Shell sh = new Shell("/bin/bash -s"); @@ -52,8 +62,8 @@ class TestHadoopExamples { private static Configuration conf; private static String HADOOP_OPTIONS; - @Before - void setUp() { + @BeforeClass + static void setUp() { conf = new Configuration(); conf.addResource('mapred-site.xml'); HADOOP_OPTIONS = @@ -80,9 +90,7 @@ class TestHadoopExamples { assertTrue("Could not create output directory", sh.getRet() == 0); } - def failures = []; - - def examples = + static Map examples = [ pi :'20 10', wordcount :"$EXAMPLES/text $EXAMPLES_OUT/wordcount", @@ -95,15 +103,29 @@ class TestHadoopExamples { randomtextwriter :"-Dtest.randomtextwrite.total_bytes=1073741824 $EXAMPLES_OUT/randomtextwriter" ]; + private String testName; + private String testJar; + private String testArgs; + + @Parameters + public static Map generateTests() { + Map res = [:]; + examples.each { k, v -> res[k] = [k.toString(), v.toString()] as Object[]; } + return res; + } + + public TestHadoopExamples(String name, String args) { + testName = name; + testArgs = args; + testJar = (name == "sleep") ? HADOOP_MR_TEST_JAR : + HADOOP_EXAMPLES_JAR; + } + @Test - void testMRExamples() { - examples.each { testName, args -> - sh.exec("$hadoop jar $HADOOP_EXAMPLES_JAR $testName $HADOOP_OPTIONS $args"); - - if (sh.getRet()) { - failures.add(testName); - } - } - assertTrue("The following tests have failed: " + failures, failures.size() == 0); + void testMRExample() { + sh.exec("$hadoop jar $testJar $testName $HADOOP_OPTIONS $testArgs"); + + assertTrue("Example $testName failed", + sh.getRet() == 0); } } Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopsmoke/TestHadoopSmoke.groovy URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopsmoke/TestHadoopSmoke.groovy?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopsmoke/TestHadoopSmoke.groovy (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopsmoke/TestHadoopSmoke.groovy Tue Nov 1 21:53:04 2011 @@ -39,7 +39,7 @@ class TestHadoopSmoke { static String testDir = "test.hadoopsmoke." + (new Date().getTime()) static String nn = (new Configuration()).get(DFSConfigKeys.FS_DEFAULT_NAME_KEY) - String cmd = "hadoop jar ${hadoopHome}/contrib/streaming/hadoop-streaming*.jar" + + String cmd = "hadoop jar ${hadoopHome}/contrib/streaming/hadoop*streaming*.jar" + " -D mapred.map.tasks=1 -D mapred.reduce.tasks=1 -D mapred.job.name=Experiment " String cmd2 =" -input ${testDir}/cachefile/input.txt -mapper map.sh -file map.sh -reducer cat" + " -output ${testDir}/cachefile/out -verbose " Modified: incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadooptests/TestTestCLI.groovy URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadooptests/TestTestCLI.groovy?rev=1196308&r1=1196307&r2=1196308&view=diff ============================================================================== --- incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadooptests/TestTestCLI.groovy (original) +++ incubator/bigtop/branches/hadoop-0.23/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadooptests/TestTestCLI.groovy Tue Nov 1 21:53:04 2011 @@ -29,7 +29,7 @@ class CommonConfigurationKeys { /** This test class serves only one purpose: to prepare environment for proper execution of parent (TestCLI) */ -class TestTestCLI extends TestCLI { +class TestTestCLI extends TestHDFSCLI { private static final String JT = "mapred.job.tracker"; static {